diff --git a/CHANGELOG.md b/CHANGELOG.md index be31ef8..a0c4de1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,63 @@ - The database was seeded once from the local dump — 1,568 chunks, 35,756 SBC chunks, 3,276 plans, 56,280 ZIP-county rows and the `bm25` index row — with zero restore errors and every count matching the source. +- Design for California plan comparison and a source registry + (`docs/superpowers/specs/2026-09-24-ca-plan-catalog-design.md`), the first of + five California sub-projects. Covered California plans come from CMS's + state-based exchange PUF, since the Marketplace API has none. Their SBC links + are blank there, and Covered California's terms forbid automated access, so + it is linked, never read. +- Implementation plan for California plan comparison + (`docs/superpowers/plans/2026-09-24-ca-plan-catalog.md`). The design is + corrected from checks against the real 2026 file: a plan is sold only where it + has a filed rate, four Los Angeles ZIPs with no CMS rating area show + unpriced, and only in-network individual cost shares are loaded. +- `src/core/exchanges.py` and `src/core/plan_year.py`: which exchange sells + each state's plans (Covered California for California, HealthCare.gov for + the 30 API states), and the plan year on sale, which is next year from + 1 November. +- A committed source registry (`src/ingestion/sources/registry.toml`, ADR + 0025): every source with its licence, permission status, `robots.txt` status + and removal steps. It is validated when loaded, only approved sources run, + and Covered California is recorded as link-only because its Terms of Use + forbid automated access. +- Migration `15bb18a57fac` for a filed-rate plan catalog: + - `plans.catalog_source`, `cms_api` or `ca_sbe_puf`, checked, so a source can + be audited or removed; + - `plan_counties.zipcodes`, for a plan sold in only some of a county's ZIPs; + - new `rating_areas`, `plan_rates` (ages 14 to 64, checked) and + `catalog_loads`, with the file's label and sha256. + API-sourced rows are unchanged. +- `make ingest-ca-plans YEAR=` (ADR 0024): loads Covered California's plans + from CMS's state-based exchange PUF, because the Marketplace API has none. + - **2026:** 190 plans from 11 issuers and 28,101 filed rates. + - **Validated first:** the whole file is checked before one transaction + writes it, so a bad file changes nothing. + - **Sold only where rated:** a plan is sold only in counties where it has a + filed rate. Western Health Advantage files one service area over two + rating areas. + - **Hand-transcribed:** rating areas and issuer names, cross-checked against + the data. Four Los Angeles ZIPs that CMS gives no rating area are reported + and shown unpriced. + - **Docs:** `docs/findings/ca-sbe-puf.md` and `docs/runbooks/california.md`. +- California plan comparison: + - **Pricing:** plan search prices California plans from CMS's filed rates for + the ZIP's rating area and the person's age, in SQL, and never calls CMS for + them. + - **Partial counties:** a plan sold in part of a county is listed only for its + ZIPs. + - **Wording:** answers say premiums are CMS's published rates before any tax + credit or state help, and name Covered California. + - **Out-of-date year:** when the plan year is no longer on sale, the answer + opens with a server-written notice saying so. It is not model-written, + because asked to write it, the model also wrote it where it was untrue. + - **Profile:** says plan comparison is available in California only once its + plans are loaded. + - **Plan table:** shows the plan year and describes each exchange's premiums + when one answer mixes states. +- README and CLAUDE.md now cover California. The scope includes Covered + California, and the README documents `make ingest-ca-plans`, the source + registry, and the PLAN SEARCH and CONCEPTS eval sections. - A `Dockerfile` for the API, and a CI job that builds and scans the image (ADR 0022) — the first of ADR 0002's two deferred items. Multi-stage, runs as a non-root user, `torch` from PyTorch's CPU index on Linux so no CUDA @@ -390,6 +447,21 @@ ### Fixed +- "Compare the gold plans." and "Compare silver plans for me." skipped the plan + search in about half of runs, in every state. The prompt left it open whether + comparing plans of a metal level meant real plans or the level in general, + and the model spent its turn deciding. The prompt now says a request to + compare, list or show plans means searching; only what a level means stays + a corpus question. + - **Before, on `main`'s prompt:** 2 of 4 runs searched. + - **After:** 35 of 35 search requests searched, and 0 of 10 concept + questions did. + - **Eval:** a new floor for concept questions that must not search. +- A Bronze plan search now includes Expanded Bronze plans, which it silently + missed; every California bronze plan is one. +- Plan answers, the "no Summary of Benefits link" reason and the plan table name + the exchange that sells the state's plans. They used to send everyone to + HealthCare.gov, which is wrong for the 21 states and DC that run their own. - The API never configured logging. Every command-line entry point calls `setup_logging()`; `create_app()` did not, so in the Flask process the root logger kept Python's default of WARNING with no handler: nothing the request diff --git a/CLAUDE.md b/CLAUDE.md index 0f1f440..8be1b0d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ PolicyPal lets a logged-in user ask natural-language questions about any kind of insurance — health, auto, life, home, and more — and returns grounded answers generated by an LLM using context retrieved from a vector store. -**Built:** comparison of real purchasable health plans from the CMS Marketplace API. That half is narrower than the Q&A — ACA individual and family plans, in the 30 states on HealthCare.gov. PolicyPal compares plans, never recommends one. All five phases in [docs/plans](docs/plans/) are complete; keeping the plan documents current is a routine, in [docs/runbooks/sbc.md](docs/runbooks/sbc.md). +**Built:** comparison of real purchasable health plans from the CMS Marketplace API, and for California from CMS's state-based exchange PUF (ADR 0024). That half is narrower than the Q&A — ACA individual and family plans, in the 30 states on HealthCare.gov plus California (Covered California). PolicyPal compares plans, never recommends one. All five phases in [docs/plans](docs/plans/) are complete; keeping the plan documents current is a routine, in [docs/runbooks/sbc.md](docs/runbooks/sbc.md). ## Stack diff --git a/Makefile b/Makefile index 5df78fd..2d36289 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: ingest build-index ingest-plans ingest-sbc refresh-sbc sbc-report migrate api \ +.PHONY: ingest build-index ingest-plans ingest-ca-plans ingest-sbc refresh-sbc sbc-report migrate api \ ui-dev ui-build ui-lint ui-preview ui-test test lint check # The plan year to work on; the commands default to the calendar year, which @@ -21,6 +21,14 @@ ingest-plans: @test -n "$(STATES)" || { echo "STATES is required, e.g. make ingest-plans STATES=TX,FL (or STATES=ALL)"; exit 2; } uv run python -m src.ingestion.plans --states $(STATES) $(YEAR_ARG) +# California's plans, from CMS's state-based exchange PUF (ADR 0024); the +# Marketplace API has none. YEAR is required: a plan year's file appears May +# to August of that year, and until then this exits 3. ZIP=path loads a file +# downloaded by hand; REFRESH=1 downloads it again. e.g. make ingest-ca-plans YEAR=2026 +ingest-ca-plans: + @test -n "$(YEAR)" || { echo "YEAR is required, e.g. make ingest-ca-plans YEAR=2026"; exit 2; } + uv run python -m src.ingestion.ca_puf --year $(YEAR) $(if $(ZIP),--zip $(ZIP)) $(if $(REFRESH),--refresh) + # Summary of Benefits PDFs for the catalog plans in STATES; run ingest-plans # first. Required for the same reason. e.g. make ingest-sbc STATES=NH,DE # TOP_ISSUERS=1 reads only the largest parent companies' plans, ISSUERS=40788,66252 diff --git a/README.md b/README.md index 8b05a5d..8ab486d 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ PolicyPal lets a logged-in user ask natural-language questions about any kind of insurance — health, auto, life, home, and more — and returns grounded answers generated by an LLM using context retrieved from a vector store. Every answer cites the source passages it was built from. -**Built:** comparison of real purchasable health plans drawn from the CMS Marketplace API. That half is deliberately narrower than the Q&A — ACA individual and family plans, in the 30 states served by HealthCare.gov. The 21 state-based exchanges, employer coverage, Medicare and Medicaid have no equivalent public catalog and are out of scope. All five phases of [docs/plans](docs/plans/) are complete; the catalog is loaded for eighteen states (3,276 plans, 137 issuers), and 2,274 of those plans (69.4%) have their Summary of Benefits read here — 2,136 read whole and 138 in part (ADR 0017). Keeping the documents current is a routine now, not a phase: [docs/runbooks/sbc.md](docs/runbooks/sbc.md). +**Built:** comparison of real purchasable health plans drawn from the CMS Marketplace API, and for California from CMS's state-based exchange data (ADR 0024). That half is deliberately narrower than the Q&A — ACA individual and family plans, in the 30 states served by HealthCare.gov plus California's Covered California (190 plans from 11 issuers for 2026, priced from CMS's filed rates). The other 20 state-based exchanges, employer coverage, Medicare and Medicaid are out of scope. All five phases of [docs/plans](docs/plans/) are complete; the catalog is loaded for eighteen states (3,276 plans, 137 issuers), and 2,274 of those plans (69.4%) have their Summary of Benefits read here — 2,136 read whole and 138 in part (ADR 0017). Keeping the documents current is a routine now, not a phase: [docs/runbooks/sbc.md](docs/runbooks/sbc.md). PolicyPal will compare plans; it will not recommend one. Advice-shaped questions are refused by design, and that refusal is a measured property with its own regression floor (ADR 0004). @@ -17,7 +17,7 @@ Because HealthCare.gov is the authoritative consumer source, answers about healt - **Conversational Q&A**, organized into threads, over a curated insurance knowledge base - **Hybrid RAG pipeline** — BM25 + pgvector semantic search, cross-encoder reranked and relevance-filtered before reaching the LLM, so weakly-relevant matches never become context - **Cited answers** — every reply lists the source passages and their relevance score, and the plans it compared in a side-by-side table; both survive reloading the conversation. Each plan in the table says whether its Summary of Benefits was read, and why not (ADR 0017) -- **Real plan comparison** — ask about ACA Marketplace plans; the model searches the ingested catalog and CMS prices the plans live for your age (ADR 0010). It compares plans and never recommends one +- **Real plan comparison** — ask about ACA Marketplace plans; the model searches the ingested catalog and CMS prices the plans live for your age (ADR 0010). In California, premiums are CMS's filed rates for your age and ZIP's rating area, and answers name Covered California (ADR 0024). It compares plans and never recommends one - **A profile, kept off the model** — signup takes a ZIP code, date of birth and county, and plan questions use them without their ever being sent to the LLM. Nobody under 13 can sign up (ADR 0012) - **JWT-authenticated API** — only a signed-in user can query, and only ever sees their own threads - **Reproducible ingestion** — a single command runs fetch → normalize → chunk → embed → store, across every registered source @@ -248,6 +248,34 @@ ingested before that table existed, re-run once. Needs `CMS_MARKETPLACE_API_KEY` explicitly: `uv run python -m src.ingestion.plans --states TX --year 2027`. The county list is cached per plan year in `data/plans/raw/`; delete it to refresh. + +#### California (Covered California) + +```bash +make ingest-ca-plans YEAR=2026 # ZIP=path for a file downloaded by hand, REFRESH=1 to re-download +``` + +The Marketplace API has no California plans, so they come from CMS's +state-based exchange PUF, one zip per plan year, published May to August of +that year (ADR 0024). The loader: + +- writes the same tables, plus `rating_areas`, `plan_rates` and `catalog_loads`; +- validates the whole file before writing, in one transaction, so a bad file + changes nothing; +- keeps a plan only in counties where it has a filed rate. + +Premiums are read from `plan_rates` in SQL; CMS is never called for California. +Rating areas and issuer names are transcribed by hand and re-checked yearly: +[`docs/runbooks/california.md`](docs/runbooks/california.md). What the file +holds is in [`docs/findings/ca-sbe-puf.md`](docs/findings/ca-sbe-puf.md). + +#### Source registry + +Every source PolicyPal reads or links is recorded in +[`src/ingestion/sources/registry.toml`](src/ingestion/sources/registry.toml), +with its licence, permission status, `robots.txt` status and removal steps +(ADR 0025). Only approved sources run. Covered California is recorded as +link-only: its Terms of Use forbid automated access. Verified API behaviour is recorded in [`docs/findings/cms-marketplace-api.md`](docs/findings/cms-marketplace-api.md). @@ -434,6 +462,13 @@ cite general material or state a figure for it. Its floor is full marks, and eve COVERAGE answer is held to the same rule. It also needs the NC catalog: `make ingest-plans STATES=NC` and `make ingest-sbc STATES=NC TOP_ISSUERS=1`. +PLAN SEARCH requires every plan question to reach `search_plans`. That includes +comparisons by metal level ("Compare the gold plans."), which used to skip it, +and two California questions that must name Covered California, never +HealthCare.gov. They need the California catalog: `make ingest-ca-plans +YEAR=2026`. CONCEPTS requires the opposite for a question about what a level +means ("What is a silver plan?"), which is answered from the corpus. + `uv run python -m scripts.eval_sbc_ranking` checks, without calling a model, that the section answering each of ten questions is among the four `plan_coverage` ranks highest, for two plans per issuer with stored SBCs. Run it before and after a parser diff --git a/docs/decisions/0024-california-plans-from-the-sbe-puf.md b/docs/decisions/0024-california-plans-from-the-sbe-puf.md new file mode 100644 index 0000000..33a3b70 --- /dev/null +++ b/docs/decisions/0024-california-plans-from-the-sbe-puf.md @@ -0,0 +1,109 @@ +# 0024 — California plans come from CMS's state-based exchange PUF, priced from filed rates + +## Status + +Accepted. Amends ADR 0009 (a second pipeline writes the plan catalog) and ADR +0010 (California premiums come from filed rates, not the live API). + +## Context + +A user with a California ZIP got "not a marketplace state", and the model sent +them to HealthCare.gov. That is the wrong exchange. California runs Covered +California, and the CMS Marketplace API carries none of its plans: +`MARKETPLACE_STATES` is the 30 HealthCare.gov states. + +The federal source that does cover California is CMS's **State-based Exchange +public use file** (), +one zip per state and plan year. The 2026 file +(`californiasbpuf2026.zip`, labelled `05052026`, data dated 2026-06-03) was +profiled before any code was written; the numbers are in +[docs/findings/ca-sbe-puf.md](../findings/ca-sbe-puf.md). + +- **Plans:** 190 individual on-exchange medical plans from 11 issuers, each with + cost-sharing variants. Every bronze plan is "Expanded Bronze". +- **Premiums:** 28,101 filed monthly rates by plan, rating area (1–19) and age + band (0-14, 15 to 63, 64 and over). There is no tobacco rating. +- **Service areas:** by issuer and service-area ID; 124 rows cover only some ZIPs + of a county, and 5 cover the whole state. +- **Missing:** issuer names, and the SBC, brochure, formulary and network URLs. + Those columns are empty on every row. +- **Rating areas:** not in the file. CMS publishes them separately, as a page + (). + Counties map to one area each, except Los Angeles, which splits by 3-digit ZIP + prefix. +- **Publication lag:** the file comes out months into its plan year: 2026 on + 2026-06-03, 2025 on 2025-05-06, 2024 on 2024-05-14. During open enrollment for + next year (1 November to 31 January in California) only this year's data exists. + +Covered California's own site is not an alternative. Its Terms of Use forbid +access "through any automated means (including, but not limited to, use of +scripts, web crawlers, or screen scrapers)" and republishing without a written +agreement. + +## Decision + +1. **A second, California-only loader** (`src/ingestion/ca_puf/`, + `make ingest-ca-plans YEAR=`). It writes the existing `issuers`, `plans`, + `plan_counties` and `plan_cost_shares`, plus three new tables: + - `rating_areas`: county, or county and ZIP prefix, to rating area; + - `plan_rates`: plan, area and age to a monthly rate; + - `catalog_loads`: which file, which bytes (sha256), when. + + `plans.catalog_source` (`cms_api` or `ca_sbe_puf`) records which pipeline + wrote each plan, so a source can be audited or removed. `MARKETPLACE_STATES` + is unchanged. A generic loader for all 22 state-based exchanges was + declined: each state still needs its own rating areas and issuer names. +2. **Premiums are filed rates, read in SQL.** The rate for the plan, the ZIP's + rating area and the person's age, clamped to 14..64, is the premium. It is + the same figure the live path shows: CMS's price for one person with no + income, before any tax credit. CMS is never called for California; + `age_rated_premiums` refuses a state it does not serve. + `premium_reference` is NULL for California plans. +3. **A plan is sold only where it has a rate.** Western Health Advantage files + one service area over rating areas 2 and 3, with a separate plan for each, + each rated in one area. Taking the service area alone would offer each plan in + counties it has no price for. So a service-area county is kept only where the + plan is rated for that county's area; the 2026 file drops 40 of 1,239 pairs. +4. **The load is validated before any write, in one transaction.** A bad file + changes nothing, and the previous load keeps serving. The load aborts when: + - an issuer is unknown; + - a county (other than Los Angeles) is missing from the map; + - a service-area county is missing from the ZIP crosswalk; + - a plan has no rates or is sold nowhere; + - a plan is rated in an area where it is not sold. + + The last check is what catches a mistyped county in the hand-transcribed map. + In the 2026 file, 156 of the 190 plans are rated in exactly one area. +5. **A ZIP with no rating area is shown unpriced, never guessed.** Four real Los + Angeles ZIPs in CMS's 2026 crosswalk have a prefix in neither of the county's + lists: 90134, 90140 and 90189 (901), and 93063 (930). Their plans are listed + with "no filed rate", and the load reports them. +6. **Only in-network, individual cost shares are loaded,** for every + cost-sharing variant. The search reads only these, and they are the only + values whose API wording is verified. The variant labels in the PUF (`Limited + Cost Sharing Plan Variation`, `94% AV Level Silver Plan`, …) are the API's. +7. **Say the year, and point to the exchange.** A search in California reports + `premium_source = cms_filed_rates`, and `prior_year` when the plan year is + before the year on sale (the next year from 1 November). The model then leads + with "these are 2026 plans; 2027 plans aren't available here yet" and names + Covered California. This stops by itself once the new file is loaded. +8. **Name the right exchange everywhere.** Tool results carry the exchange for + the state. The prompts, the SBC "no link" reason and the frontend no longer + send every state to HealthCare.gov. A Bronze filter now includes Expanded + Bronze, in every state. + +## Consequences + +- California users compare real Covered California plans, with a premium for + their age and ZIP, matching CMS's file to the cent. That was spot-checked + against the raw rates. +- **California data lags every open enrollment by about six months.** This is + said in every answer where it applies; it is not hidden. +- **Rating areas and issuer names are hand-transcribed.** They are re-checked + every year ([docs/runbooks/california.md](../runbooks/california.md)), and the + load's cross-checks fail loudly if they drift. +- **SBC documents are not loaded yet.** Every California plan shows "no Summary + of Benefits link", because the PUF has none. Sub-project 2 of the California + expansion builds a hand-reviewed list of links instead. +- **Quality ratings are not in the PUF.** California plans show "Not available", + not "Not rated". diff --git a/docs/decisions/0025-a-committed-source-registry.md b/docs/decisions/0025-a-committed-source-registry.md new file mode 100644 index 0000000..7bd3199 --- /dev/null +++ b/docs/decisions/0025-a-committed-source-registry.md @@ -0,0 +1,65 @@ +# 0025 — Every source is recorded in a committed registry, and only approved ones run + +## Status + +Accepted. Records in data what ADR 0003 (corpus sources) and ADR 0013 (SBC +copyright) decided in prose. + +## Context + +PolicyPal is a pilot that may become commercial. Its sources differ in licence: +- Wikipedia is CC BY-SA; +- HealthCare.gov and CMS data are US government works (17 U.S.C. § 105); +- issuer SBCs are copyrighted mandated disclosures; +- Covered California reserves all rights and forbids automated access. + +Those decisions lived in ADR prose and code comments. A commercial launch has to +be able to list every source, its licence and permission status, and how to +take it out. Nothing stopped a new source from running without that being +written down. + +## Decision + +`src/ingestion/sources/registry.toml` holds one `[[source]]` per source, +enabled or refused, read with the standard library's `tomllib`, so there is no +new dependency: + +| Field | Values | +| --- | --- | +| `kind` | `corpus`, `catalog`, `sbc_host`, `directory` | +| `jurisdiction` | `US` or a state code | +| `permission_status` | `public_domain`, `open_license`, `written_permission`, `pending_review`, `denied` | +| `commercial_use` | `yes`, `no`, `review` | +| `robots` | `allowed`, `disallowed`, `unavailable`, `not_applicable`; `robots_checked_on` is required unless `not_applicable` | +| `access` | `api`, `download`, `crawl`, `manual`, `link` | + +Every entry also has `id`, `name`, `publisher`, `scope_urls`, `license`, +`license_url`, `verified_on`, `enabled`, `removal` and `notes`. + +`src/ingestion/sources/registry.py` enforces the rules when the registry loads: +- required fields are present, and no unknown field; +- values are from the lists above; +- ids are unique; +- `enabled = true` needs `public_domain`, `open_license` or `written_permission`. + +An invalid registry fails the import, so a bad edit cannot quietly enable +anything. + +- `SOURCES`, the corpus sources the pipeline runs, is filtered at import to + those the registry enables. +- `ALL_SOURCES` lists them all; a test requires each to have an entry. +- The California plan loader calls `require_enabled("cms_ca_sbe_puf")` before + fetching. +- A refused source is recorded too, disabled, with the reason. Covered + California is `denied` with `access = "link"`, so the decision is not + re-litigated. + +## Consequences + +- A new source cannot run without an entry, and its removal steps are written + down when it is added. +- The registry is a record, not a permission. A commercial launch still needs + legal review of every enabled entry. `commercial_use = "yes"` records what + the licence says, not a legal opinion. +- Entries go stale: `verified_on` and `robots_checked_on` say when each was last + checked, and the California runbook re-checks its entries yearly. diff --git a/docs/design/01-ca-plan-catalog-design.md b/docs/design/01-ca-plan-catalog-design.md new file mode 100644 index 0000000..fdf4b5a --- /dev/null +++ b/docs/design/01-ca-plan-catalog-design.md @@ -0,0 +1,461 @@ +# California plan comparison, and a source registry — design + +**Status:** proposed · **Date:** 2026-09-24 · **Sub-project:** 1 of 5 in the California expansion +(SP0 + SP1). Later sub-projects are SBC documents, SBC hybrid search, California rules and rights, +and a provider-directory spike. Each gets its own spec. + +## Why + +A signed-up user with a California ZIP gets `not_marketplace_state`, and the model tells them to +go to HealthCare.gov (`src/services/generation.py:76-77`). That is the wrong exchange. California +runs Covered California, and the CMS Marketplace API has none of its plans. + +This change makes plan comparison work for California: the same cited, compare-never-recommend +table the 30 HealthCare.gov states get. It also starts a committed source registry. The expansion +is a pilot today and may become commercial, so every data source needs a recorded license and +permission status, and a way to be removed. + +## Scope + +**In:** +- ACA individual and family plans sold on Covered California, plan year 2026. +- Premiums by age, deductibles and out-of-pocket maximums, filtered by county and ZIP. +- The exchange named correctly wherever the app names one. +- A source registry covering the existing sources and the new ones. + +**Out, each for a later sub-project:** +- SBC documents and plan coverage answers (SP2). +- Cross-plan coverage search (SP3). +- California rules and rights content (SP4). +- Provider networks (SP5). +- Off-exchange and small-group (SHOP) plans. +- Dental plans. +- Subsidy estimates: the app shows the full, pre-subsidy price, as it does today. +- Other states that run their own exchanges. The loader is California-only by decision. + +## The data, as verified + +**Plans come from the CMS State-based Exchange public use file (PUF)** +- Source: `https://www.cms.gov/files/zip/californiasbpuf2026.zip`, a federal work in the public domain. +- It was downloaded and profiled on 2026-09-24. The file is labelled `05052026`, and CMS dates the + data 2026-06-03. + +Individual, on-exchange, medical-only plans: + +| Item | Value | +| --- | --- | +| Plan IDs | 663: 190 base plans (`STANDARD COMPONENT ID`), each with CSR variants | +| CSR variant suffix | `-01` standard, `-02` zero cost sharing, `-03` limited cost sharing, `-04`/`-05`/`-06` Silver 73/87/94 | +| Issuers | 11. `ISSUER NAME` is blank on every row | +| Metal levels | Silver, Gold, Platinum, **Expanded Bronze**, Catastrophic. No plain "Bronze" | +| Rates | age `0-14`, 15–63, `64 and over` × `Rating Area 1`–`19`; tobacco `No Preference` | +| Service areas | keyed by **(issuer, service area ID)**, since IDs such as `CAS001` repeat across issuers. 124 partial-county rows list ZIPs; 5 statewide rows have a blank county | +| Link columns | SBC, brochure, formulary and network URLs are **empty on every row** | +| Money text | `"$5,200 "`, `"Not Applicable"`, `""`, `"$10600 per person \| $21200 per group"` | +| Integrated MOOP | `MEDICAL DRUG MAXIMUM OUT OF POCKET INTEGRATED` is `Yes` on every row, so only the TEHB MOOP columns carry values | + +The rates file also covers dental and small-group plans, so the loader filters by the plan IDs it +kept. + +**Issuer names.** The file doesn't give them. They are curated and were checked against each +issuer's service-area footprint: + +| HIOS issuer ID | Carrier | +| --- | --- | +| 27603 | Anthem Blue Cross | +| 70285 | Blue Shield of California | +| 47579 | Balance by CCHP (San Francisco, San Mateo) | +| 67138 | Health Net | +| 51396 | Inland Empire Health Plan (Riverside, San Bernardino) | +| 40513 | Kaiser Permanente | +| 92815 | L.A. Care Health Plan (Los Angeles) | +| 18126 | Molina Healthcare | +| 92499 | Sharp Health Plan (San Diego) | +| 84014 | Valley Health Plan (Santa Clara) | +| 93689 | Western Health Advantage | + +These match the 11 carriers Covered California announced for 2026, after Aetna's exit. Before +merging, cross-check the names against a federal issuer list, such as the CMS rate-review or MLR +public use files. + +**Rating areas** come from CMS, "California Geographic Rating Areas" +(`https://www.cms.gov/cciio/programs-and-initiatives/health-insurance-market-reforms/ca-gra`), read +from the raw page. +- All 58 counties map to areas 1–14 and 17–19. +- Los Angeles County splits by 3-digit ZIP: + - area 15: 906, 907, 908, 910, 911, 912, 915, 917, 918, 935 + - area 16: 900, 902, 903, 904, 905, 913, 914, 916, 923, 928, 932 +- Some real Los Angeles ZIPs have a prefix in neither list. In CMS's 2026 crosswalk that is 4 of + 508: 90134, 90140 and 90189 (prefix 901), and 93063 (930, a ZIP shared with Ventura County). + 901 also appears in the PUF's own partial-county ZIP lists. Such a ZIP has no rating area: its + plans show as unpriced, never at a guessed price, and the load reports them. +- Checked against the 2026 data: the 57 non-Los Angeles counties plus Los Angeles are exactly the + crosswalk's California counties. Every plan's rated areas are areas where it is sold, and no + plan ends up sold nowhere. + +**Publication lag.** CMS publishes each year's SBE PUF months into that year: 2026 on 2026-06-03, +2025 on 2025-05-06, 2024 on 2024-05-14. During 2027 open enrollment (2026-11-01 to 2027-01-31) only +2026 California data exists. + +## Decisions + +1. **A second catalog pipeline, not a generic one.** A California-only loader writes into the + existing plan tables. `MARKETPLACE_STATES` stays exactly as it is, and the 30-state API path is + untouched. California joins as a *filed-rate* state: its prices are stored, not fetched live. +2. **Prices come from filed rates, in SQL.** + - The premium for a person is the PUF rate for their plan, rating area and age band. + - This is the same figure the live path shows: CMS's `POST /plans` with a single-person household + and no income returns the full, pre-subsidy price. + - `premium_reference` stays NULL for California, so a missing rate means unpriced, never free + (ADR 0010). +3. **Label the year and degrade (user decision).** + - Show the newest California year loaded, labelled with its plan year. + - When that year is before the plan year on sale (the next year from 1 November), the answer + leads with: these are {year} plans and prices; {year on sale} plans are not available here + yet; see Covered California. + - When CMS publishes the new file and it is loaded, this switches off with no code change. +4. **Covered California is linked, never read.** Its Terms of Use forbid automated access and + republishing without a written agreement. The app names it and links to it; nothing fetches it. +5. **A committed source registry**, before any new source is used (user decision: commercial + later). +6. **Fix the Bronze filter for every state.** A `Bronze` filter currently matches only `Bronze` + (`src/services/plan_search.py:201-202`), so every California bronze plan, and every Expanded + Bronze plan in the API states, is missed. `Bronze` now matches both. + +## Design + +### SP0 — source registry and exchanges + +**`src/ingestion/sources/registry.toml`** is committed and read with the standard library's +`tomllib`, so there is no new dependency. There is one `[[source]]` table per source: + +| Field | Values | +| --- | --- | +| `id` | stable key, e.g. `cms_ca_sbe_puf` | +| `name`, `publisher`, `scope_urls` | descriptive | +| `kind` | `corpus` · `catalog` · `sbc_host` · `directory` | +| `jurisdiction` | `US` or a state code | +| `license`, `license_url` | as published | +| `permission_status` | `public_domain` · `open_license` · `written_permission` · `pending_review` · `denied` | +| `commercial_use` | `yes` · `no` · `review` | +| `robots`, `robots_checked_on` | `allowed` · `disallowed` · `unavailable` · `not_applicable` | +| `access` | `api` · `download` · `crawl` · `manual` | +| `verified_on` | ISO date | +| `enabled` | bool; only `public_domain`, `open_license` or `written_permission` may be `true` | +| `removal` | how to take this source out | +| `notes` | free text | + +Entries in this change: + +| id | Enabled | +| --- | --- | +| `wikipedia` | yes | +| `healthcare_gov` | yes | +| `cms_marketplace_api` | yes | +| `cms_ca_sbe_puf` | yes | +| `cms_ca_rating_areas` | yes | +| `coveredca` | **no**: `denied`, link only, with the Terms of Use clause cited | + +The CA carrier SBC hosts, CDI and DMHC are recorded in the sub-projects that would use them. + +**`src/ingestion/sources/registry.py`** +- `load_registry()` +- `require_enabled(source_id)` raises `SourceNotApprovedError` +- `validate()` checks required fields, the allowed values, and that no enabled entry has an + unapproved permission status. + +`Source` (`src/ingestion/sources/base.py`) gains a `registry_id`. The corpus pipeline and the new +loader call `require_enabled` before fetching. A test asserts that every `Source` in +`src/ingestion/sources/__init__.py` has an entry. + +**`src/core/exchanges.py`** (in core, because ingestion and services both use it): + +```python +@dataclass(frozen=True) +class Exchange: + name: str + url: str + +HEALTHCARE_GOV = Exchange("HealthCare.gov", "https://www.healthcare.gov/") +FILED_RATE_STATES = {"CA": Exchange("Covered California", "https://www.coveredca.com/")} +CATALOG_STATES = MARKETPLACE_STATES + tuple(FILED_RATE_STATES) + +def exchange_for(state) -> Exchange | None: ... # HealthCare.gov for API states, CA's own, else None +``` + +**`src/core/plan_year.py`**: `plan_year_on_sale(on: date) -> int` returns `on.year + 1` from +1 November, otherwise `on.year`. + +### SP1 — schema + +One Alembic migration after head `f6e50c4aec26`: + +| Change | Type | Notes | +| --- | --- | --- | +| `plans.catalog_source` | `String(16)` NOT NULL, server default `'cms_api'` | California rows get `'ca_sbe_puf'`. Makes a source auditable and removable | +| `plan_counties.zipcodes` | `ARRAY(String(5))`, nullable | NULL means the whole county. That is what the API path writes, so it is unaffected | +| `rating_areas` (new) | `id`, `state`, `plan_year`, `countyfips`, `zip3 String(3)` NOT NULL default `''`, `rating_area SmallInteger` | Unique `(plan_year, countyfips, zip3)`. `''` means the whole county: not NULL, because NULLs are distinct in a unique key (see `PlanCostShare`) | +| `plan_rates` (new) | `id`, `plan_id` FK → `plans.id` ON DELETE CASCADE, `rating_area SmallInteger`, `age SmallInteger`, `individual_rate Numeric(10,2)` | Unique `(plan_id, rating_area, age)`. `age` 14 stands for the 0–14 band, 64 for 64 and over | +| `catalog_loads` (new) | `id`, `source`, `state`, `plan_year`, `file_url`, `file_label`, `sha256`, `plans`, `loaded_at` | One row per load. The data-freshness label comes from here | + +The models go in `src/models/plan.py`. `alembic check` must stay clean. + +### SP1 — loader: `src/ingestion/ca_puf/` + +| Module | Job | +| --- | --- | +| `download.py` | Fetches the zip to `data/plans/raw/ca_puf/{year}.zip` (gitignored), written aside and renamed, as in `src/ingestion/marketplace_api.py:41-67`. Uses `USER_AGENT` from `src/ingestion/constants.py`. HTTP 404 means "not published yet", exit code 3. `--zip PATH` takes a file downloaded by hand | +| `read.py` | Pure functions from the zip to row dicts. No database, so it is fully unit-testable | +| `issuers.py` | `CA_ISSUERS`, the table above, with its source and verified date. An unknown issuer ID raises | +| `rating_areas.py` | The county → area map and the Los Angeles ZIP3 split, with the CMS URL and verified date | +| `load.py` | Writes one state-year in one transaction | +| `__main__.py` | `python -m src.ingestion.ca_puf --year 2026 [--zip PATH]` | + +**Reading (`read.py`)** +- CSVs are read with `csv.DictReader` and `utf-8-sig`. +- Rows kept: `MARKET COVERAGE=Individual`, `DENTAL ONLY PLAN=No`, `QHP NONQHP TYPE ID=On the + Exchange`, and a plan ID suffix other than `-00`. +- **Plan rows** use the base plan ID from `STANDARD COMPONENT ID`. The other fields come from the + `-01` row: + - `marketing_name` = `PLAN MARKETING NAME` + - `plan_type` = `PLAN TYPE` + - `metal_level` verbatim, so `Expanded Bronze` stays as CMS writes it (ADR 0009) + - `hsa_eligible` and `has_national_network` from Yes/No + - `state` = `CA`, `catalog_source` = `ca_sbe_puf` + - `is_standardized_plan`, the quality ratings, `premium_reference` and the four URL fields are + `None` +- **Cost shares** give one `plan_cost_shares` row per non-empty value. + + The search reads only the combination no CSR, `In-Network`, `Individual` (`_cost_shares()`, + `src/services/plan_search.py:161-185`). Those rows must use exactly the strings the API path + writes, which are verified in `docs/findings/cms-marketplace-api.md`: + + | PUF columns (INN TIER 1, individual) | `kind` | `cost_share_type` | + | --- | --- | --- | + | `TEHB DED INN TIER 1 INDIVIDUAL` | deductible | `Combined Medical and Drug EHB Deductible` | + | `MEHB DED INN TIER1 INDIVIDUAL` | deductible | `Medical EHB Deductible` | + | `DEHB DED INN TIER1 INDIVIDUAL` | deductible | `Drug EHB Deductible` | + | `TEHB INN TIER 1 INDIVIDUAL MOOP` | moop | `Maximum Out of Pocket for Medical and Drug EHB Benefits (Total)` | + + - Suffix `-01` maps to `csr_variant = "Exchange variant (no CSR)"`, `network_tier = "In-Network"`, + `family_cost = "Individual"`. + - The other CSR variants (`-02` to `-06`) are stored with `CSR VARIATION TYPE` as the variant, + e.g. `Limited Cost Sharing Plan Variation`, the same wording the API uses. They are stored for + in-network tier 1, individual only. + - Tier 2, out-of-network and family values are **not** loaded. Nothing reads them, and the API's + words for them are unverified, so storing them would mean inventing labels. +- **Money parsing** is its own `_puf_money()`: it trims, drops `$` and `,`, and returns None for + `""` and `Not Applicable`. In the 2026 file, the six columns loaded hold only `$…` values or + `""`. + `plans._money` expects numbers, so it can't be reused. +- **Service areas** are joined on (`ISSUER ID`, `SERVICE AREA ID`). + - The county FIPS is the part of `COUNTY` after ` - `. + - A partial county becomes a sorted list of its ZIPs. + - Duplicate plan-county pairs are merged, and a whole-county row wins over a partial one. + - `COVER ENTIRE STATE=true` expands to every California county in `zip_counties`. +- **Rates** are keyed by (base plan ID, area number, age key), from `INDIVIDUAL RATE`. +- **Where a plan is sold.** A plan is sold in a service-area county only where it has a rate for + that county's rating area. + - Western Health Advantage files one service area across areas 2 and 3, with separate plan IDs + per area, each rated in one area. So 40 of 1,239 pairs in the 2026 file are dropped. + - In Los Angeles, a plan's ZIP list keeps the ZIPs whose area it is rated in, plus ZIPs with no + area. It becomes NULL (the whole county) when that is every Los Angeles ZIP. + +**Loading (`load.py`)** +1. Require `zip_counties` rows for the year. If there are none, write them by reusing + `county_zips(year)` and `_write_zip_counties` (`src/ingestion/plans.py:198`). That needs the CMS + key; without it, stop and say to run `make ingest-plans` first. +2. **Validate before writing.** Abort with nothing written if any of these hold: + - a California county other than Los Angeles is missing from the rating-area map; + - a service-area county is missing from `zip_counties`; + - an issuer is unknown; + - a plan has no rates, or ends up sold in no county; + - a plan is rated in an area where it is not sold. A mistyped county in the map fails here. + 156 of the 190 plans are rated in exactly one area. +3. Upsert issuers and plans with `plans.upsert` (`src/ingestion/plans.py:141`). +4. Replace the California plans' counties, cost shares and rates for the year, and delete + California plans for the year that are no longer in the file. The file is authoritative. +5. Replace `rating_areas` for the year, and insert a `catalog_loads` row with the zip's sha256. + +The whole load is one transaction: a failure leaves the previous load intact. + +**CLI:** `make ingest-ca-plans YEAR=2026 [ZIP=path]`. `resolve_states` (`src/ingestion/plans.py:43`) +keeps `make ingest-plans` limited to the 30 API states. Its error for `CA` names +`make ingest-ca-plans`. + +### SP1 — search (`src/services/plan_search.py`) + +- **Where a ZIP is served.** Split out `_resolve_place(session, zip_code, county_fips)` from + `search_plans` (lines 271-293); SP3 will reuse it. A county is served when its state is in + `MARKETPLACE_STATES`, or is in `FILED_RATE_STATES` **and** has plans loaded. Before any California + load, a California ZIP still gets `not_marketplace_state`, now carrying the Covered California + exchange. +- **The year.** `_counties_for_zip` already picks the newest year in which the ZIP's own county has + plans (lines 128-148). A California ZIP stays on 2026 after federal 2027 data loads, with no change + needed. +- **ZIP-aware service areas.** The loaded-county check and `find_plans` add + `PlanCounty.zipcodes IS NULL OR :zip = ANY(PlanCounty.zipcodes)`. `find_plans` gains a + `zip_code` argument. +- **Bronze.** A `Bronze` filter becomes `metal_level IN ('Bronze', 'Expanded Bronze')`. An + `Expanded Bronze` filter stays exact. +- **Filed-rate pricing**, when the county's state is in `FILED_RATE_STATES`: + - `_rating_area(session, year, countyfips, zip_code)` takes the ZIP3 row, otherwise the + whole-county row. It returns None for a Los Angeles ZIP with an unlisted prefix, and those plans + are unpriced. + - `rate_age(age) = min(max(age, 14), 64)`. + - `find_plans` outer-joins `plan_rates` on (plan, area, age), orders by that rate when sorting by + premium, and fills in `monthly_premium` and `premium_age`. + - `_price` is not called. + - `age_rated_premiums` raises `ValueError`, before any HTTP request, for a state outside + `MARKETPLACE_STATES`: defence in depth. +- **Catastrophic plans:** the existing age rule (line 295) applies unchanged. +- **`PlanSearchResult`** gains: + - `exchange`: an `Exchange` or None; + - `premium_source`: `cms_live` or `cms_filed_rates`; + - `plan_year_on_sale`; + - `prior_year`: `plan_year < plan_year_on_sale(profile.today())`, only for filed-rate states. + +### SP1 — tools, prompts, profile, frontend + +**Tools and prompts** +- `tools._render` (`src/services/tools.py:189`) adds `exchange` (name and URL), `premium_source`, + `prior_year` and `plan_year_on_sale`. `not_marketplace_state` carries `exchange` when one is known. +- `generation.PLAN_TOOL_PROMPT`: + - `not_marketplace_state` points to the exchange named in the result, or "the state's own + exchange" when none is named. HealthCare.gov is never hard-coded. + - For `cms_filed_rates`: the premium is CMS's published {plan_year} rate for that plan and age, + before any federal tax credit or California premium help. + - For `prior_year`: lead with "These are {plan_year} plans and prices; {plan_year_on_sale} plans + aren't available here yet", then point to the exchange. +- `COVERAGE_PROMPT` (line 131) and `sbc_status.REASONS["no_link"]` stop naming HealthCare.gov. Until + SP2, every California plan is `no_link`, and the wording must hold for it. + +**Profile.** `src/services/profile.py`: +- `is_marketplace_state` becomes `plan_search_available(state)`, meaning in `CATALOG_STATES`. +- The API gains no fields. The frontend already has each county's and each saved plan's state, and + looks up the exchange in its own small map. Two maps, one per side, are pinned by a test on each + side, which is cheaper than threading an exchange through every payload and the stored plan cards. +- `marketplace_state` keeps its name, for compatibility, but now means "plan comparison available". +- No new endpoints. + +**Frontend** +- `frontend/src/features/plans/exchanges.js` maps a state to its exchange name and URL. +- `PlanComparison.jsx`: + - the caption says "{plan_year} plans"; + - the footnote links the right exchange, and for California says the premiums are CMS's published + rates before any tax credit or state help; + - the quality column shows "Not available" for California instead of "Not rated". +- `ProfileFields.jsx` names Covered California for a California county. + +## Error handling + +| Case | Behaviour | +| --- | --- | +| PUF not yet published (404) | Exit code 3 with a message; nothing written | +| PUF columns renamed or missing | `read.py` raises naming the column, before any write | +| Validation failure (unmapped county, unknown issuer, plan sold nowhere or rated where not sold) | The transaction is not started; the previous load stays | +| California ZIP, nothing loaded | `not_marketplace_state` with Covered California | +| Rate missing for a plan, area or age at query time, or a ZIP with no rating area | Unpriced, never free, and never a guessed price | +| Live CMS call attempted for California | Raises before HTTP; caught as today, so the plan shows unpriced | + +## Security + +- Parameterized SQLAlchemy only. +- The ZIP and age are used in SQL filters, never logged. The existing log lines stay as they are. +- The download goes over HTTPS to a fixed CMS host. +- No PUF file is committed; `data/` stays gitignored. +- No new endpoints and no new fields. `marketplace_state` is now true for California. The + authentication and rate limits of `/api/profile` and `/api/counties` are unchanged, and re-tested. + +## Testing + +**Pytest** +- `tests/test_ca_puf_read.py`, using small synthetic CSVs in `tests/fixtures/ca_puf/`, made up and + not copied from the real file: + - money formats; + - suffix → variant; + - the four row filters; + - Expanded Bronze kept; + - an unknown issuer raises; + - the service-area join on the issuer pair; + - statewide expansion and partial ZIP lists; + - age keys; + - the Los Angeles split, with 909 raising. +- `tests/test_ca_puf_load.py`, against the database: + - loading twice gives the same counts; + - a plan dropped from the file is deleted; + - `catalog_loads` is written; + - Texas plans are untouched; + - a rating-area mismatch aborts with nothing written. +- `tests/test_plan_search.py`: + - California prices from `plan_rates` with no CMS call; + - Los Angeles areas 15 and 16; + - a ZIP outside a partial county; + - ages 10 and 70; + - `prior_year` with today patched to 2026-11-15, and false once a 2027 load exists; + - an unloaded California ZIP gets Covered California; + - `Bronze` matches Expanded Bronze; + - catastrophic plans excluded at 30; + - the Texas path unchanged. +- `tests/test_marketplace_api.py`: California raises without HTTP. +- `tests/test_tools.py` and `tests/test_generation.py`: the new fields render, and there is no + unconditional HealthCare.gov. +- `tests/test_profile_api.py`: California available with its exchange; New York unavailable with + none; the `/api/profile` and `/api/counties` 429 limits re-asserted. +- `tests/test_registry.py`: + - every `Source` is registered; + - only approved entries are enabled; + - required fields are present; + - `coveredca` is disabled. + +**Vitest:** `PlanComparison`, `ProfileFields`, `useProfileFields` and `exchanges`, covering rendering, +the California and Texas footnotes, the year caption, "Not available", and accessibility names on +the new links. + +**Playwright** (the plugin, run against the local stack): +1. Sign up with ZIP 90012. +2. The profile names Covered California. +3. "Compare silver plans" returns a table with a 2026 caption, the filed-rate footnote and the + Covered California link. + +**Eval:** add California cases to `scripts/eval_generation.py` `run_plan_searches`: a Los Angeles +Silver request and a Bronze request. They check that Covered California is named and HealthCare.gov +is not. + +**CI:** `make check`, which runs lint, tests with coverage ≥85%, and `alembic check`. + +## Documentation + +- **ADR 0024**, California plans from the SBE PUF. It amends ADR 0009 (a second catalog pipeline; + `premium_reference` NULL for California) and ADR 0010 (California prices come from filed rates in + SQL). +- **ADR 0025**, a committed source registry. +- `docs/findings/ca-sbe-puf.md`: the profile of the data above, and the cost-share vocabulary + caveat. +- `docs/runbooks/california.md`, the yearly routine: + 1. From May to August, watch for the next year's PUF. + 2. Re-verify the rating-area page and the issuer list. + 3. Run `make ingest-ca-plans YEAR=`. + 4. Dump and restore to Azure (`docs/runbooks/deploy.md` step 5). +- README: scope now includes Covered California plans. CHANGELOG entries for each commit. + +## Delivery + +Two PRs from `feature_ca_plan_catalog`, or a stacked second branch: + +1. **SP0:** the registry, exchanges, plan year on sale, the Bronze fix and the HealthCare.gov + wording fix. This is useful on its own and changes no data. +2. **SP1:** the migration, loader, filed-rate search, profile and frontend, and data loaded locally. + Before merging, the evidence: + - 190 plans; + - 11 issuers; + - every California plan-county has a rate; + - the Playwright run. + +After merging, seed Azure per the deploy runbook. + +## Open items + +- Cross-check the curated issuer names against a federal issuer list (a pre-merge task, SP1). +- Blue Shield publishes a rating-region page (`blueshieldca.com/regions`) that could serve as a + second check of the county map. It is used only as a check, never as a source. diff --git a/docs/findings/ca-sbe-puf.md b/docs/findings/ca-sbe-puf.md new file mode 100644 index 0000000..57aa0f2 --- /dev/null +++ b/docs/findings/ca-sbe-puf.md @@ -0,0 +1,127 @@ +# CMS state-based exchange PUF — California + +What the file actually holds, checked against the real data before and after +the loader was written (ADR 0024). This is a record of observations on the +dates given, not a spec: CMS can change the file. + +## Where it comes from + +- **Page:** . + It covers 22 state-based exchanges, California among them, for plan years 2016–2026. +- **File:** `https://www.cms.gov/files/zip/californiasbpuf{year}.zip`, a US + government work. cms.gov's `robots.txt` (checked 2026-09-24) disallows neither + `/files/` nor `/cciio/`. +- **Publication dates:** 2026 data is dated 2026-06-03, 2025 is 2025-05-06 and + 2024 is 2024-05-14. There is **no 2027 file during 2027 open enrollment**. +- **The 2026 zip** (downloaded 2026-09-24): sha256 + `8a6d606a6da66b6d61296416d720f0d6b7a2766b398f52757f6ebe3600c52225`, 1,025,734 + bytes. It holds `CAPlans05052026.csv`, `CARates…`, `CAServiceAreas…`, + `CABenefits…`, `CANetworks…` and `CABusinessRules…`. The CSVs are plain ASCII + with no byte-order mark. + +## Plans (`CAPlans`) + +**Shape** +- 1,010 rows: individual and small group (SHOP), medical and dental. +- **Kept:** `MARKET COVERAGE = Individual`, `DENTAL ONLY PLAN = No` and + `QHP NONQHP TYPE ID = On the Exchange`. That leaves 663 plan IDs, which are + **190 base plans** (`STANDARD COMPONENT ID`) with their variants. +- **Variant suffixes:** + + | Suffix | `CSR VARIATION TYPE` | + | --- | --- | + | `-01` | `Standard On Exchange Plan` | + | `-02` | `Zero Cost Sharing Plan Variation` | + | `-03` | `Limited Cost Sharing Plan Variation` | + | `-04` | `73% AV Level Silver Plan` | + | `-05` | `87% AV Level Silver Plan` | + | `-06` | `94% AV Level Silver Plan` | + + There is no `-00` (off-exchange) row among the kept ones. + +**Plan fields** +- **Metal levels** among the 190: Gold 46, Platinum 45, Silver 45, **Expanded + Bronze 33**, Catastrophic 21. There is no plain "Bronze". +- **Plan types:** HMO, PPO, EPO. +- `ISSUER NAME` is **blank on every row**. +- `URL FOR SUMMARY OF BENEFITS COVERAGE`, `PLAN BROCHURE` and `FORMULARY URL` are + **empty on every row**, and so is `NETWORK URL` in `CANetworks`. + +**Cost sharing** +- `MEDICAL DRUG MAXIMUM OUT OF POCKET INTEGRATED` is `Yes` everywhere, so only the + `TEHB … MOOP` columns hold values. +- The six in-network tier 1 individual columns hold only `$…` values (e.g. + `"$5,200 "`, with a trailing space) or `""`. +- **Loaded:** 1,932 cost-share rows (four columns × variants, non-empty only). + +**Issuers:** 11, named by hand in `src/ingestion/ca_puf/issuers.py`. Each HIOS ID +was matched by its network names and service-area footprint, e.g. 84014 sells +only in Santa Clara (Valley Health Plan), and 47579 only in San Francisco and San +Mateo (CCHP). This matches the 11 carriers Covered California announced for 2026; +Aetna left. + +## Rates (`CARates`) + +- 93,840 rows, including dental and small-group plans. **28,101** belong to the + 190 kept plans. That is 551 plan-area pairs × 51 age bands (`0-14`, 15 to 63, + `64 and over`), with no duplicate or conflicting rate. +- `TOBACCO` is `No Preference` throughout: California does not rate tobacco. +- **156 of the 190 plans are rated in exactly one rating area.** +- **Spot check:** premiums shown by plan search for ZIPs 90012 and 90601 match + `INDIVIDUAL RATE` to the cent. Examples: + - L.A. Care Bronze 60 HMO, age 40: area 16 $367.43, area 15 $346.63. + - Kaiser Silver 70 HMO, age 40, area 16: $481.83. + +## Service areas (`CAServiceAreas`) + +- Keyed by (`ISSUER ID`, `SERVICE AREA ID`). IDs such as `CAS001` repeat across + issuers. +- **307 individual medical rows:** + - 124 partial counties, with their ZIPs listed; + - 5 `COVER ENTIRE STATE = true` rows, with a blank county. +- `COUNTY` looks like `Los Angeles - 06037`. All 58 California counties appear. + +## Rating areas (not in the PUF) + +The source is CMS's page +. +It was transcribed into `src/ingestion/ca_puf/rating_areas.py` and checked +against the raw page on 2026-09-24. + +- 57 counties map to areas 1–14 and 17–19. +- Los Angeles splits by ZIP prefix: + - area 15: 906, 907, 908, 910, 911, 912, 915, 917, 918, 935 + - area 16: 900, 902, 903, 904, 905, 913, 914, 916, 923, 928, 932 +- **909 is in neither list.** A web summary had put it in area 15. +- **Four Los Angeles ZIPs in CMS's 2026 crosswalk have an unlisted prefix:** + 90134, 90140 and 90189 (901), and 93063 (930, which spans Ventura County). The + PUF's own partial-county lists include 901 ZIPs. These ZIPs have no area: their + plans load and show unpriced. + +## Where plans are sold + +**Western Health Advantage (93689)** files one service area, `CAS001`, over +Marin, Napa, Solano and Sonoma (area 2) and Sacramento, Placer, El Dorado and +Yolo (area 3). Each plan is rated in one area only: `…0110001`–`…0120005` in +area 2 and `…0150001`–`…0160003` in area 3. Taking the service area alone would +offer each plan in four counties it has no rate for. + +With "sold only where it has a rate for the county's area", the 2026 file +gives: +- **1,199 plan-county pairs kept, 40 dropped** (all Western Health Advantage); +- no plan sold nowhere; +- no plan rated in an area where it is not sold; +- all 58 counties served. + +The loader enforces the last two, which a mistyped county in the map would break. + +## Load of 2026 (2026-09-24, local database) + +``` +190 plans from 11 issuers, file 05052026: 1199 plan-county rows (40 dropped: no rate there), 28101 rates +Los Angeles ZIPs with no CMS rating area, shown unpriced: 90134, 90140, 90189, 93063 +``` + +- `rating_areas`: 78 rows for 2026 (57 counties plus 21 Los Angeles prefixes). +- 310 plan-county rows list ZIPs. +- Reloading changes no row count; the 3,276 API-sourced plans are untouched. diff --git a/docs/runbooks/california.md b/docs/runbooks/california.md new file mode 100644 index 0000000..bbb17fa --- /dev/null +++ b/docs/runbooks/california.md @@ -0,0 +1,73 @@ +# Runbook — California plans (Covered California) + +California's plans come from CMS's state-based exchange PUF, not the +Marketplace API (ADR 0024). The file for a plan year appears **May to August +of that year**, so this is a yearly routine, not a monthly one. + +## Prerequisites + +- `docker compose up -d db` and `make migrate`. +- A ZIP-to-county crosswalk for the plan year. `make ingest-ca-plans` fetches + CMS's if it is missing, which needs `CMS_MARKETPLACE_API_KEY`. Any + `make ingest-plans` run for that year also writes it. + +## Every year, May to August + +1. **Look for the file.** + + ```sh + make ingest-ca-plans YEAR=2027 + ``` + + Exit status 3 with "CMS has not published the 2027 California PUF yet" means + try again in a few weeks. Nothing was written. +2. **Re-check the rating areas** against + . + Update `src/ingestion/ca_puf/rating_areas.py`, with the date, if anything + moved. +3. **Re-check the issuers.** + - Covered California's rates announcement names the year's carriers. The + 2026 one came out on 2025-08-14. + - A new HIOS issuer ID fails the load with `issuer … is not in CA_ISSUERS`. + Add it to `src/ingestion/ca_puf/issuers.py` with where the name came from. + Never guess. +4. **Read the load's summary**, and compare it with the previous year's in + [docs/findings/ca-sbe-puf.md](../findings/ca-sbe-puf.md): + + ``` + plans from issuers, file