diff --git a/CHANGELOG.md b/CHANGELOG.md index a0c4de1..bc7b0c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,72 @@ ### Added +- Design and ADR 0026 for California's SBCs, sub-project 2 + (`docs/superpowers/specs/2026-09-25-ca-sbc-design.md`). CMS's California + file has no SBC links, so each plan's link comes from a hand-built manifest. + **Website Terms govern:** a carrier whose Terms ban robots or scrapers is + manual-only, even where `robots.txt` allows us, as Covered California already + was. ADR 0026 amends ADRs 0013, 0019 and 0025. +- The source registry records the 11 California carriers as `sbc_host` + entries, each with its Terms, `robots.txt` outcome, access and removal + steps: + - **crawled:** IEHP, Molina, Sharp and Balance by CCHP; + - **manual-only:** Kaiser, Blue Shield, Western Health Advantage, Anthem, + L.A. Care and Valley; + - **disabled:** Health Net, whose SBCs sit behind a search form. + + They use a new status, `mandated_disclosure`: an SBC is published because + federal law requires it, which is not a licence. So it is allowed only for an + `sbc_host`, and its commercial use stays `review`. `sbc_host_for` finds a + link's entry by the longest matching prefix, and every scope URL must now be + an `https://host/…` prefix, so it can't also match a look-alike host. +- California's SBCs are linked from a committed, hand-built manifest, + `src/ingestion/sbc/manifests/ca-2026.csv`: 114 of 190 plans, 58 documents. + - `make apply-sbc-manifest YEAR=` checks every row first: plan ID, year, a + safe HTTPS link, and a California registry entry covering it. Then it sets + each California plan's link, and a plan with no row, or whose carrier is + disabled, gets none. `make ingest-ca-plans` applies it after each load. + - `make check-sbc-manifest YEAR=` lists, for a person to check: + - documents whose printed title lacks the plan's name; + - failures; + - plans awaiting a manual import; + - unread links; + - plans with no link. +- `make ingest-sbc STATES=CA` and `make sbc-report STATES=CA`. California links + are read only as their carrier's entry allows: + - `crawl` is fetched; + - `manual` is never requested, and is counted as awaiting import; + - disabled or unregistered is skipped. + + `STATES=ALL` is still the 30 HealthCare.gov states. First run: the 19 IEHP, + Molina and Sharp SBCs all read `ok`, with no name mismatches. +- `make import-sbc YEAR= DIR=` (or `FILE= URL=`) reads an SBC a person + downloaded in a browser. + - **Checks:** the link must be a plan's and its carrier enabled, and the file + a PDF within the size cap. + - **What it does:** it is copied to the link's cache path, and any different + file there is archived. It is then parsed like a crawled one, with no + request. `DIR=` matches each file to the one manual-only link with its + name, or its `?fileName=` value. + - **Afterwards:** the document is recorded as `manual` and never re-requested. + `refresh-sbc` skips it; after a parser change it is re-parsed from disk, + and a missing file is reported for re-import. +- Migration `3b1d6e2a9c47`: `sbc_documents.acquisition`, `crawl` or `manual`, + checked, default `crawl`. `sbc-report` counts read documents by it. +- California coverage answers say that a plan's SBC is its standard version, + so people who qualify for cost-sharing reductions or American Indian and + Alaska Native cost sharing pay less than it shows. The server writes the + notice, like the prior-year one. A first version asked the model to repeat + a note from the tool result, and in a live question it left it out. +- `docs/findings/ca-sbc.md`: where each California carrier publishes its SBCs, + what its Terms and `robots.txt` say (quoted), how each manifest row was + checked, what scripts requested before the Terms were read, and the first + run's counts. The eval still passes: 1,303 of 1,320 ranking questions put + the right section in the top 4 (98.7%; floor 97%). +- Runbooks: `california.md` gains the yearly SBC routine (re-check Terms, write + the manifest, apply, crawl, import, check), and `sbc.md` how manual documents + behave in the monthly one. The README documents the new commands and the + registry's `mandated_disclosure`. - **PolicyPal is deployed to Azure.** The API runs on Container Apps in Central US on the image digest built from `main`, backed by a PostgreSQL 16 Flexible Server with `pgvector`, and the browser app is on Static Web Apps. @@ -447,6 +513,9 @@ ### Fixed +- `make ingest-sbc` counted a recorded failure twice: once as "already + current" and again as a "recorded failure skipped". So a run over blocked + documents read as healthier than it was. It now counts it once, as skipped. - "Compare the gold plans." and "Compare silver plans for me." skipped the plan search in about half of runs, in every state. The prompt left it open whether comparing plans of a metal level meant real plans or the level in general, diff --git a/Makefile b/Makefile index 2d36289..5fd1c8d 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,5 @@ -.PHONY: ingest build-index ingest-plans ingest-ca-plans ingest-sbc refresh-sbc sbc-report migrate api \ +.PHONY: ingest build-index ingest-plans ingest-ca-plans ingest-sbc refresh-sbc sbc-report apply-sbc-manifest \ + check-sbc-manifest import-sbc migrate api \ ui-dev ui-build ui-lint ui-preview ui-test test lint check # The plan year to work on; the commands default to the calendar year, which @@ -29,7 +30,8 @@ ingest-ca-plans: @test -n "$(YEAR)" || { echo "YEAR is required, e.g. make ingest-ca-plans YEAR=2026"; exit 2; } uv run python -m src.ingestion.ca_puf --year $(YEAR) $(if $(ZIP),--zip $(ZIP)) $(if $(REFRESH),--refresh) -# Summary of Benefits PDFs for the catalog plans in STATES; run ingest-plans +# Summary of Benefits PDFs for the catalog plans in STATES; run ingest-plans (for +# CA, ingest-ca-plans and its SBC manifest: ADR 0026) # first. Required for the same reason. e.g. make ingest-sbc STATES=NH,DE # TOP_ISSUERS=1 reads only the largest parent companies' plans, ISSUERS=40788,66252 # only those HIOS issuers' (ADR 0015). Downloaded PDFs are always kept (ADR 0016). @@ -49,6 +51,27 @@ refresh-sbc: sbc-report: uv run python -m src.ingestion.sbc.report $(YEAR_ARG) $(if $(STATES),--states $(STATES)) $(if $(VERIFY),--verify-files) +# California's SBC links, from the committed src/ingestion/sbc/manifests/ca-YEAR.csv +# (ADR 0026). ingest-ca-plans applies it too; run this after editing the file +# or the source registry. e.g. make apply-sbc-manifest YEAR=2026 +apply-sbc-manifest: + @test -n "$(YEAR)" || { echo "YEAR is required, e.g. make apply-sbc-manifest YEAR=2026"; exit 2; } + uv run python -m src.ingestion.sbc.manifest apply --year $(YEAR) + +# After `make ingest-sbc STATES=CA`: documents whose printed title doesn't +# name their plan, failures, and what still awaits a manual import. Reads only. +check-sbc-manifest: + @test -n "$(YEAR)" || { echo "YEAR is required, e.g. make check-sbc-manifest YEAR=2026"; exit 2; } + uv run python -m src.ingestion.sbc.manifest check --year $(YEAR) + +# An SBC a person downloaded in a browser, for a carrier whose documents are +# manual-only (ADR 0026). FILE=x.pdf URL=, or DIR=folder +# to match files to links by file name. e.g. make import-sbc YEAR=2026 DIR=~/Downloads/ca-sbc +import-sbc: + @test -n "$(YEAR)" || { echo "YEAR is required, e.g. make import-sbc YEAR=2026 DIR=folder"; exit 2; } + @test -n "$(FILE)$(DIR)" || { echo "FILE= with URL=, or DIR=, is required"; exit 2; } + uv run python -m src.ingestion.sbc.import_pdf --year $(YEAR) $(if $(FILE),--file "$(FILE)" --url "$(URL)") $(if $(DIR),--dir "$(DIR)") + migrate: uv run alembic upgrade head diff --git a/README.md b/README.md index 8ab486d..c2660d6 100644 --- a/README.md +++ b/README.md @@ -275,7 +275,9 @@ Every source PolicyPal reads or links is recorded in [`src/ingestion/sources/registry.toml`](src/ingestion/sources/registry.toml), with its licence, permission status, `robots.txt` status and removal steps (ADR 0025). Only approved sources run. Covered California is recorded as -link-only: its Terms of Use forbid automated access. +link-only: its Terms of Use forbid automated access. Each California carrier +whose SBCs are used is an `sbc_host` entry with status `mandated_disclosure`, +which is not a licence, so its commercial use stays under review (ADR 0026). Verified API behaviour is recorded in [`docs/findings/cms-marketplace-api.md`](docs/findings/cms-marketplace-api.md). @@ -318,6 +320,30 @@ sections into `sbc_chunks`, kept apart from the corpus that `make ingest` rebuil (ADR 0019), so ingesting another state does not re-request every blocked host. - For a quick check: `uv run python -m src.ingestion.sbc --states NH --limit 5`. +#### California's SBCs + +```bash +make apply-sbc-manifest YEAR=2026 # links from src/ingestion/sbc/manifests/ca-2026.csv +make ingest-sbc STATES=CA YEAR=2026 # reads only the carriers that may be crawled +make import-sbc YEAR=2026 DIR=~/Downloads/ca-sbc # PDFs a person downloaded in a browser +make check-sbc-manifest YEAR=2026 # what still needs a person's eye +``` + +CMS's California file has no SBC links. A committed, hand-built manifest gives +each plan its link, and each carrier has a registry entry (ADR 0026). + +- **Crawled:** carriers whose `robots.txt` and website Terms both allow + automated access. +- **Manual only:** carriers whose Terms ban robots or scrapers, and hosts that + block bots. A person downloads their SBCs in a browser, and `make import-sbc` + reads the file with no request to the carrier. The document is recorded as + `manual` and never re-requested. + +California answers say the SBC is the plan's standard version, since +cost-sharing reductions lower what it shows. The yearly steps are in +[docs/runbooks/california.md](docs/runbooks/california.md), and the carrier +research in [docs/findings/ca-sbc.md](docs/findings/ca-sbc.md). + ### Keeping documents current ```bash diff --git a/docs/decisions/0013-sbc-documents.md b/docs/decisions/0013-sbc-documents.md index 388182a..3a787df 100644 --- a/docs/decisions/0013-sbc-documents.md +++ b/docs/decisions/0013-sbc-documents.md @@ -10,6 +10,8 @@ Amended by ADR 0018: chart rows are rebuilt from the table's ruled grid, one line per service, and a PDF with no text layer is refused as scanned. Its "a refused `robots.txt` counts as disallowing everything" is **superseded by ADR 0020**, which follows RFC 9309's reading of an unavailable robots.txt. +Amended by ADR 0026 for California: a carrier's website Terms can refuse us +too, and a document a person downloads in a browser may be imported by hand. ## Context diff --git a/docs/decisions/0019-keeping-sbcs-current.md b/docs/decisions/0019-keeping-sbcs-current.md index 8db8d93..4a46db9 100644 --- a/docs/decisions/0019-keeping-sbcs-current.md +++ b/docs/decisions/0019-keeping-sbcs-current.md @@ -4,7 +4,8 @@ Accepted. Phase 4. Supersedes ADR 0015's "Known limits, left to Phase 4", and amends ADR 0013's "a failure is retried on the next run" and ADR 0016's -folders. +folders. Amended by ADR 0026: a document imported by hand is never +re-requested, by `ingest-sbc` or `refresh-sbc`. ## Context diff --git a/docs/decisions/0025-a-committed-source-registry.md b/docs/decisions/0025-a-committed-source-registry.md index 7bd3199..50382af 100644 --- a/docs/decisions/0025-a-committed-source-registry.md +++ b/docs/decisions/0025-a-committed-source-registry.md @@ -3,7 +3,9 @@ ## Status Accepted. Records in data what ADR 0003 (corpus sources) and ADR 0013 (SBC -copyright) decided in prose. +copyright) decided in prose. Amended by ADR 0026: a fourth approved status, +`mandated_disclosure`, for `sbc_host` entries only, and every scope URL must +be an `https://host/…` prefix. ## Context @@ -28,7 +30,7 @@ new dependency: | --- | --- | | `kind` | `corpus`, `catalog`, `sbc_host`, `directory` | | `jurisdiction` | `US` or a state code | -| `permission_status` | `public_domain`, `open_license`, `written_permission`, `pending_review`, `denied` | +| `permission_status` | `public_domain`, `open_license`, `written_permission`, `mandated_disclosure` (ADR 0026: `sbc_host` only, `commercial_use = "review"`), `pending_review`, `denied` | | `commercial_use` | `yes`, `no`, `review` | | `robots` | `allowed`, `disallowed`, `unavailable`, `not_applicable`; `robots_checked_on` is required unless `not_applicable` | | `access` | `api`, `download`, `crawl`, `manual`, `link` | @@ -40,7 +42,10 @@ Every entry also has `id`, `name`, `publisher`, `scope_urls`, `license`, - required fields are present, and no unknown field; - values are from the lists above; - ids are unique; -- `enabled = true` needs `public_domain`, `open_license` or `written_permission`. +- `enabled = true` needs `public_domain`, `open_license`, `written_permission` + or, since ADR 0026, `mandated_disclosure`; +- each `scope_urls` entry is an `https://host/…` prefix (ADR 0026), so it + cannot also match a look-alike host. An invalid registry fails the import, so a bad edit cannot quietly enable anything. diff --git a/docs/decisions/0026-california-sbcs-manifest-terms-and-manual-import.md b/docs/decisions/0026-california-sbcs-manifest-terms-and-manual-import.md new file mode 100644 index 0000000..b647caa --- /dev/null +++ b/docs/decisions/0026-california-sbcs-manifest-terms-and-manual-import.md @@ -0,0 +1,208 @@ +# 0026 — California's SBCs: a hand-built manifest, website Terms, and manual import + +## Status + +Accepted. Sub-project 2 of the California expansion. It amends three earlier +ADRs: + +- **ADR 0013.** For California, a carrier's website Terms can refuse us, as well + as its `robots.txt` and HTTP replies. A person downloading a document in a + browser is not the crawler working around a refusal. +- **ADR 0019.** A document imported by hand is never re-requested, not even by + `refresh-sbc`. +- **ADR 0025.** Adds the permission status `mandated_disclosure`, and enabling + an entry now accepts it. + +## Context + +California's plans come from CMS's state-based exchange PUF (ADR 0024). Its SBC +URL column is empty on every row, so every California coverage question ended +with "no Summary of Benefits and Coverage link is listed for this plan". The +SBC pipeline could not help as it stood: + +- it accepted only the 30 HealthCare.gov states; +- it read only `plans.benefits_url`, which CMS fills for those states; +- it had no way to take a PDF a person had downloaded. + +The 11 carriers were researched one by one on 2026-09-25 and 2026-09-26. The +evidence is in [docs/findings/ca-sbc.md](../findings/ca-sbc.md). In short: + +- **They publish SBCs in very different ways:** + - named by HIOS ID (Kaiser); + - named by metal level (IEHP); + - by internal codes; + - only behind a search form (Health Net). + + Their pages print no HIOS IDs, so there is no reliable way to match a + document to a plan automatically. +- **Most carriers' website Terms restrict what we do, even where `robots.txt` + allows us.** + - Kaiser, Blue Shield, Western Health Advantage and Anthem expressly ban + robots, spiders or scrapers; L.A. Care appears to as well. + - Nearly all of them limit use to personal, non-commercial purposes, or bar + reproduction without written consent. + + The registry already records Covered California as `denied` with the note + "robots.txt allows everything; the Terms govern". +- **One carrier's `robots.txt` also names AI crawlers** (L.A. Care), and one + host blocks automated clients outright (Valley Health Plan). + +## Decision + +### A hand-built, committed manifest + +`src/ingestion/sbc/manifests/ca-.csv` lists +`hios_plan_id,plan_year,sbc_url,verified_on,note`. A person finds and checks +each row. It holds public URLs and plan IDs only, never a document or an +excerpt. An automatic "proposal helper", which would have scraped listing pages +and matched names, was dropped: most carriers' pages don't suit it, and several +carriers' Terms forbid it. + +`src/ingestion/sbc/manifest.py`: + +- **`load_manifest`** checks the file without a database: + - the header; + - the California plan ID format; + - the year; + - that the URL is HTTPS on a public host (`fetch.unsafe_reason`); + - that a California `sbc_host` registry entry covers the URL; + - the date; + - no plan listed twice. + + Any failure names the line, and nothing is applied. +- **`apply_manifest` is authoritative for California.** + - Every `ca_sbe_puf` plan of the year gets its row's URL, or NULL when it has + no row or its host's entry is disabled. + - A plan in the manifest that isn't loaded is an error. + - Disabling a carrier's entry and applying again is how a carrier is removed. +- **Where it runs.** `make apply-sbc-manifest YEAR=` runs it, and so does + `make ingest-ca-plans` after a load, when the year's manifest exists. A bad + manifest leaves the plans loaded and exits non-zero, saying the links were + not applied. +- **`make check-sbc-manifest YEAR=` is the human check after an ingest.** It + lists: + - documents whose printed title lacks a word of the plan's name; + - failures; + - plans awaiting a manual import; + - links not read yet; + - plans with no link. + +### Each carrier is a registry entry, with a new status + +Each California carrier whose SBCs are used has one `sbc_` entry of +`kind = "sbc_host"`. Its `scope_urls` are exact URL prefixes, and the longest +matching prefix wins (`registry.sbc_host_for`). The registry now requires every +prefix, of any kind, to be `https://host/…`, so a prefix can't also match a +look-alike host. + +Its `permission_status` is the new **`mandated_disclosure`**. It means "federal +law requires the issuer to give this document to anyone shopping", which is +ADR 0013's basis for using SBCs at all. It is **not** a licence, and validation +enforces both of these: + +- only an `sbc_host` may use it; +- its `commercial_use` must be `review`. + +A commercial launch needs a legal review of every one of these entries. + +### Website Terms govern, as for Covered California + +A carrier is crawled only when **neither** its `robots.txt` **nor** its website +Terms refuse automated access. + +- **Where the Terms ban robots or scrapers, the entry is `access = "manual"`.** + This applies to Kaiser, Blue Shield, Western Health Advantage, Anthem and + L.A. Care. It also applies to Valley Health Plan, whose host blocks automated + clients. No automated request is made to these hosts at all, including probes + and listing pages. +- **Where neither refuses, the entry is `access = "crawl"`,** read by the normal + fetcher with all of ADR 0013's checks. This applies to IEHP, Molina, Balance + by CCHP and Sharp. Sharp's SBC folder isn't under the path its `robots.txt` + disallows. +- **Health Net is recorded but disabled.** Its SBCs are only reachable through + a search form, and its plans keep "no link" and point to Covered California. + +### California links are gated at the start of the crawl + +`make ingest-sbc STATES=CA` now works: `resolve_states` takes the allowed +list, and the SBC ingest and report pass `CATALOG_STATES`. `ALL` still means +the 30 API states. + +For a plan in a filed-rate state, `ingest.execute` sorts each link by its +host's entry: + +| Entry | What happens | +| --- | --- | +| enabled, `crawl` | fetched | +| enabled, `manual` | never requested; counted as awaiting `make import-sbc` | +| disabled or missing | skipped and counted | + +The HealthCare.gov states' links come from CMS and are read as before. +Extending the registry gate, and this Terms rule, to them is a follow-up, not +part of this decision. + +### Manual import: a person downloads, the pipeline only reads + +`make import-sbc YEAR= FILE= URL=`, or `DIR=`. `DIR=` matches each file to the +one manual-only link whose file name it has. + +1. **Checks, before anything is written:** + - the URL is some plan's link that year; + - its host's entry is enabled; + - the file is a PDF (`fetch.looks_like_pdf`) of at most `MAX_BYTES`. +2. **Copy.** Any cached file with other bytes is archived (ADR 0016). The file + is copied to the link's cache path; the person's own file is left where it is. +3. **Parse.** `ingest_document(..., acquisition="manual")` parses it exactly as + a crawled one: the cached copy is served and no request is made. + +A person opening a carrier's page and saving its SBC is what any shopper does, +and what the document exists for. So it is not the crawler working around a +refusal, which ADR 0013 still forbids. Where the Terms limit use to personal +or non-commercial purposes, that limit is recorded as `commercial_use = +"review"` and stands until a legal review. + +### Provenance + +`sbc_documents.acquisition` is `crawl` or `manual`, NOT NULL, default `crawl`. + +- `_store` writes it only when given, so a re-parse keeps `manual`. +- A manual document is never re-requested. + - `ingest-sbc` re-parses it from disk after a parser change. + - If its file is missing, `ingest-sbc` asks for a re-import instead of a + fetch. + - `refresh-sbc` never asks the carrier about it. +- `sbc-report` counts documents read by each route. + +### Answers say the SBC is the standard plan's + +A California plan's SBC is its standard (`-01`) version. People who qualify +for cost-sharing reductions, or American Indian and Alaska Native cost +sharing, pay less than it shows. + +- When `plan_coverage` quotes a passage from a filed-rate state's plan, the + answer opens with a notice saying so. The server writes it, with the same + mechanism as the prior-year notice (`ToolOutcome.notice`), so it appears + exactly where it is true. +- A first version put a `note` in the tool result and told the model to + include it. In a live question the model left it out, so that approach was + dropped. + +Per-variant SBCs are a follow-up. + +## Consequences + +- **Answers depend on a person, yearly.** Most California carriers' SBCs come + from someone downloading them in a browser. The California runbook adds this + to the yearly routine, and `check-sbc-manifest` shows what is outstanding. +- **Commercial use is not settled.** Nearly every carrier limits use to + personal or non-commercial purposes. A pilot can proceed on a mandated + disclosure, but a launch needs each `mandated_disclosure` entry reviewed, + and possibly written permission. +- **The 30-state crawl is now inconsistent with this rule.** It reads, for + example, Anthem's SBCs for other states, and Anthem's Terms ban robots. This + is recorded here and left for a follow-up decision, not changed silently. +- **A wrong manifest row shows up as a name mismatch,** not as an error. The + check lists it and a person decides; carriers print plan names their own way. +- **Removing a carrier is two steps:** disable its entry, then run + `apply-sbc-manifest`. Its documents and PDFs stay (ADR 0016), and no plan + links to them any more. diff --git a/docs/findings/ca-sbc.md b/docs/findings/ca-sbc.md new file mode 100644 index 0000000..c3a6bd3 --- /dev/null +++ b/docs/findings/ca-sbc.md @@ -0,0 +1,181 @@ +# Findings — California's SBC documents (2026) + +Research for sub-project 2 of the California expansion (ADR 0026), done +2026-09-25 to 2026-09-26. CMS's California PUF has no SBC links, so each +carrier was researched on its own. This covers: + +- where each carrier publishes its SBCs; +- what its `robots.txt` and website Terms allow; +- how each manifest row was checked; +- what was read. + +Quotes from the Terms are verbatim, taken from the pages' own HTML, except for +L.A. Care's, which were never fetched. + +## Per carrier + +| Carrier (HIOS) | Plans | Access | Why | +| --- | ---: | --- | --- | +| IEHP (51396) | 5 | crawl | No website Terms found; `robots.txt` allows everything | +| Molina (18126) | 5 | crawl | Terms have no robots clause; `robots.txt` allows everything | +| Sharp (92499) | 9 | crawl | Terms have no robots clause; the SBC folder is outside the disallowed `/docs/default-source/pdfs/` | +| Balance by CCHP (47579) | 12 | crawl, **no 2026 links** | Only 2025 SBCs are published | +| Kaiser (40513) | 7 | manual | Terms ban robots | +| Blue Shield (70285) | 60 | manual | Terms ban robots | +| Western Health Advantage (93689) | 10 | manual | Terms ban robots | +| Anthem Blue Cross (27603) | 11 | manual | Terms ban robots | +| L.A. Care (92815) | 5 | manual | `robots.txt` names AI crawlers; Terms appear to ban robots | +| Valley Health Plan (84014) | 5, **2 linked** | manual | Host blocks bots; 3 of 5 SBCs not found | +| Health Net (67138) | 61 | disabled | SBCs only behind a POST search form | + +Every entry has permission `mandated_disclosure` and `commercial_use = +"review"`. Nearly every carrier's Terms limit use to personal or non-commercial +purposes, or bar reproduction without written consent. + +### What the Terms say about automated access + +- **Kaiser** + (, version 2.2, + September 2026): + - It is a material breach to use any "'deep-link', 'page-scrape', 'robot', + 'spider', data mining tools, data gathering and extraction tools, or other + automatic device, program, algorithm or methodology, to (1) access, + acquire, copy or monitor any portion of the Site". + - Use is "solely for your personal, noncommercial use". +- **Blue Shield** + (): you may not + "Attempt to access or search the site or content with any engine, software, + tool, agent, device or mechanism other than software and/or search agents + provided by Blue Shield of California or other generally available third + party web browsers". +- **Western Health Advantage** + (): "You may not ... (2) + use any data mining, robots, or similar gathering and extraction tools". Its + reproduction ban exempts "files containing Benefit Plans and Sales + Materials". That may cover SBCs, but it is not relied on. +- **Anthem** (): + - You will not "(8) create a database by systematically downloading and + storing this Application". + - Nor "(9) use any robot, spider, site search/retrieval application or other + manual or automatic device to retrieve, index, "scrape," "data mine" or in + any way gather information from this Application". +- **L.A. Care** (): known only from + search results, never fetched. These appear to ban "any robot, spider, + scraper, or other automated means" and commercial use without written + authorization. +- **Molina** + (): + there is no robots clause, but it says "You may use the Sites solely for the + purposes of learning about (i) Molina Healthcare's health plans or (ii) other + Molina Healthcare products or services." +- **Balance by CCHP** (): there + is no robots clause. Content "may not be stored except for personal and + non-commercial use. Republication and redissemination of Content is + expressly prohibited without the prior written consent of Balance." +- **Sharp** + (): + - There is no robots clause, only "not to misuse, abuse, or overuse beyond + reasonable amounts, this website". + - "Any reproduction or redistribution of this material beyond personal use + is prohibited without the express written consent of Sharp Health Plan." +- **IEHP:** its only terms page covers its text-message service. The site + footer reads "All Rights Reserved"; IEHP is a public entity. +- **Health Net:** robots are named only in a clause against flooding or + disrupting the site. Downloads are "only for your personal, non-commercial + informational purpose". +- **Valley Health Plan:** unknown. Both valleyhealthplan.org and the county file + host (files.santaclaracounty.gov) answer automated clients with a Cloudflare + 403, `robots.txt` included. + +## Where each carrier publishes, and how its rows were checked + +- **IEHP.** Static PDFs are named by metal, e.g. `2026-Silver-70-SBC.pdf`, and + listed at . The + Silver 73/87/94 files are cost-sharing variants and are not used. Rows were + checked by crawl: each prints "Inland Empire Health Plan: HMO", for + 2026. +- **Molina.** Files are named `CA26SBCE__1.pdf`, beside the EOCs its + member-forms page lists (the page lists only the EOCs). Checked by crawl: + each prints "Molina Healthcare of California: HMO", for 2026. +- **Sharp.** Files are listed in the "2026 summary of benefits for Covered + California" section of its 2026 plan-details page. **Each SBC prints its own + HIOS plan ID** (`-01`), and every row matched. The 26793–26801 files are the + off-exchange set. +- **Balance by CCHP.** On 2026-09-26 its media library held 2026 EOCs + (`PY2026-EOC-OnX-IFP-*`) but only 2025 SBCs, which print coverage + 1/1/2025–12/31/2025. So no 2026 row is written. Its two plan sets differ + only by service area (San Francisco and San Mateo). +- **Kaiser.** Files are named by HIOS variant, e.g. + `40513CA0380003-01-en-2026.pdf`. All 7 `-01` files exist. +- **Blue Shield.** + - There is no static PDF URL. The IFP document page + () opens + `…/memberwebapp/unauth-document-download?fileName=`, and that is the + manifest link. + - The Covered California files carry "(for Covered California)" and + `ML`/`MI` form numbers. + - **One Trio HMO SBC per metal serves all 18 regional plan IDs.** In the PUF, + `70285CA80N000n` is rated only in rating area `n`, and nothing else differs + between them. + - The broker page's "Summary of Benefits" PDFs are California EOC summaries, + not federal SBCs, and the parser rejects them. +- **Western Health Advantage.** + - SBCs are generated from a query URL on `/shop/?sp=plan-sbc`. The `IE*` + product codes are the Covered California ones, and `IM*` are off-exchange. + With the wrong code, one line differs ("Minimum Value Standards"). + - Each metal's two plan IDs (rating areas 2 and 3) share one SBC. +- **Anthem.** The SBC posting site lists 52 CA + Individual 2026 rows, but shows no URLs. The manifest's deep links follow + the `dpsdeeplink` pattern the 30-state crawl already reads, built from each + row's plan name, contract code and document ID. They are **inferred and + unverified** until downloaded. +- **L.A. Care.** Five `la4307_lacc_sbc_*_en_202508.pdf` links were found in + search results; one snippet shows coverage for 2026. **Unverified** until + downloaded. The `la5600_laccd_*` files are its off-exchange line. +- **Valley Health Plan.** Gold 80 and Bronze 60 links are on the county file + host, from search results, and **unverified**. The Platinum 90, Silver 70 and + Minimum Coverage 2026 SBCs were not found; only "Schedule of Benefits" + matrices turned up. Those three plans have no link. + +### Requests made before the Terms were read + +Kaiser, Blue Shield, WHA and Anthem were contacted by scripts before their +Terms had been read and the rule was set. Recorded for completeness: + +- **Kaiser:** a ranged request for each of the 7 SBCs, and one full download + on 2026-09-25. +- **Blue Shield:** its IFP document page, rendered once in a headless browser. + No PDF was fetched. +- **Western Health Advantage:** its SBC pages, and the generated 2026 SBCs + used to check the `IE*` codes. +- **Anthem:** the SBC posting site's search. No PDF was fetched. + +None of these files is imported. `make import-sbc` reads only what a person +downloads in a browser. + +## What was read (2026-09-26) + +```text +make apply-sbc-manifest YEAR=2026 +114 California plans for 2026 linked to 58 SBC documents; 76 not in the manifest, so shown with no link + +make ingest-sbc STATES=CA YEAR=2026 +58 SBC documents behind the CA plans for 2026; 0 already current, reading 19 +39 awaiting a manual import (their host is manual-only) +19 ok +``` + +- **The 19 crawled documents all read `ok`:** 25 or 26 sections each, 6 to 14 + pages. `check-sbc-manifest` found no name mismatches. +- **`sbc-report STATES=CA`:** 19 of 190 plans read (10.0%), 95 awaiting import + and 76 with no link. `VERIFY=1` found no missing or changed file. +- **`eval_sbc_ranking --year 2026`:** 1,303 of 1,320 in the top 4 (98.7%; + floor 97%). IEHP, Molina and Sharp are 20/20 each. +- **A live question:** "What do these two plans charge for an urgent care + visit?", asked about Sharp Silver 70 Performance HMO and Kaiser Silver 70 HMO. + - Sharp is answered from its SBC ($50 copay, deductible does not apply), + cited. + - Kaiser, awaiting import, links its PDF. + - The answer opens with the standard-version notice. + +The manual carriers' counts are added here after their import. diff --git a/docs/runbooks/california.md b/docs/runbooks/california.md index bbb17fa..b088f9b 100644 --- a/docs/runbooks/california.md +++ b/docs/runbooks/california.md @@ -53,6 +53,57 @@ of that year**, so this is a yearly routine, not a monthly one. Keep the previous year loaded. Its SBC answers and saved plan cards still refer to it. +## SBC documents, every year after the load + +The PUF has no SBC links, so California's come from a committed, hand-built +manifest: `src/ingestion/sbc/manifests/ca-.csv` (ADR 0026). Each row's +host must have an `sbc_host` entry in `src/ingestion/sources/registry.toml`. + +1. **Re-check each carrier's entry.** Before any automated request, re-read + its `robots.txt` **and** its website Terms. If the Terms now ban robots or + scrapers, set `access = "manual"`, whatever `robots.txt` says. Update + `robots_checked_on` and `verified_on`. +2. **Write the new year's manifest.** Start from last year's file. + - For a `crawl` carrier, find each plan's SBC on its listing page. + - For a `manual` carrier, open its page in a browser. Never use a script. + - Set `verified_on` on each row. A new plan ID with no row is shown with no + link. [docs/findings/ca-sbc.md](../findings/ca-sbc.md) records where each + carrier publishes. +3. **Apply it, then read what can be crawled:** + + ```sh + make apply-sbc-manifest YEAR=2027 + make ingest-sbc STATES=CA YEAR=2027 + ``` + + `make ingest-ca-plans` applies the manifest itself when the file exists. The + ingest counts the links "awaiting a manual import" and never requests them. +4. **Download the manual carriers' SBCs in a browser,** into one folder. Keep + each file's name exactly as the link's last path segment. Then: + + ```sh + make import-sbc YEAR=2027 DIR=~/Downloads/ca-sbc-2027 + make import-sbc YEAR=2027 FILE=path/to/file.pdf URL= # when names differ + ``` + + `DIR=` imports only a file whose name matches exactly one manual-only link, + and reports the rest. +5. **Check it:** + + ```sh + make check-sbc-manifest YEAR=2027 + make sbc-report STATES=CA YEAR=2027 VERIFY=1 + ``` + + Open each name mismatch in the check: carriers print plan names their own + way, so a mismatch is not always wrong. Fix any row that points at another + plan's document, apply it again, and re-import. Record the counts in the + findings file. + +**Removing a carrier:** set its entry's `enabled = false`, then +`make apply-sbc-manifest YEAR=…`. Its plans lose their link. Its documents and +PDFs are kept (ADR 0016), and no plan points at them any more. + ## If a load fails It changes nothing: validation runs before the first write, and the previous diff --git a/docs/runbooks/sbc.md b/docs/runbooks/sbc.md index f4774dd..2b38374 100644 --- a/docs/runbooks/sbc.md +++ b/docs/runbooks/sbc.md @@ -66,6 +66,23 @@ uv run python -m scripts.eval_sbc_ranking --year 2026 # before and after Check `make sbc-report` shows 0 documents on an older parser, and that the file count and modification times are unchanged. +## California, and documents imported by hand + +California's links come from a manifest, and several carriers are manual-only +(ADR 0026). The yearly steps are in +[docs/runbooks/california.md](california.md). For the monthly routine: + +- **`make ingest-sbc STATES=CA` reads only what can be crawled.** It prints + how many links are "awaiting a manual import" and never requests them. +- **`make refresh-sbc` never asks a carrier about an imported document.** To + pick up a carrier's corrected SBC, download it again in a browser and re-run + `make import-sbc`. The old file is archived. +- **After a parser change,** imported documents are re-parsed from disk like + any other. If one's file is missing, the ingest says "import them again"; + it does not fetch it. +- **`make sbc-report`** shows how many read documents were crawled and how + many were imported by hand. + ## Disk, and the one copy - A kept PDF averages about 0.75 MB. `make sbc-report` prints the totals for diff --git a/docs/superpowers/specs/2026-09-25-ca-sbc-design.md b/docs/superpowers/specs/2026-09-25-ca-sbc-design.md new file mode 100644 index 0000000..f906454 --- /dev/null +++ b/docs/superpowers/specs/2026-09-25-ca-sbc-design.md @@ -0,0 +1,215 @@ +# California SBC documents — design (sub-project 2) + +**Branch:** `feature_ca_sbc` · **ADR:** 0026 · **Master plan:** the California expansion, SP2. + +## Problem + +Every California plan loaded from the CMS PUF (ADR 0024) has `benefits_url` +NULL, because the PUF's SBC column is empty. So every California coverage +question ends in "no Summary of Benefits and Coverage link is listed for this +plan". The SBC pipeline (ADR 0013, 0015, 0019) can't help as it stands: + +- it accepts only the 30 HealthCare.gov states; +- it reads only `benefits_url`; +- it has no way to take a PDF a person downloaded. + +## What was verified (2026-09-25) + +| Carrier (HIOS) | CA plans | What the site gives | `robots.txt` for our agent | +| --- | ---: | --- | --- | +| Kaiser (40513) | 7 | SBCs named by HIOS variant, e.g. `…/2026/40513CA0380003-01-en-2026.pdf`. The listing page needs cookies; the PDFs don't. | allowed | +| Blue Shield (70285) | 60 | The broker page's PDFs are California "Summary of Benefits" (part of the EOC), **not** the federal SBC. The parser rejects them. Federal SBCs are elsewhere, still to be located. | allowed (`*/download(s)/` refused) | +| Health Net (67138) | 61 | SBCs only behind a POST search form. | allowed | +| IEHP (51396) | 5 | Static PDFs named by metal level. | allows everything | +| Anthem (27603) | 11 | — | `Disallow: /*.pdf$` | +| L.A. Care (92815) | 5 | — | names AI crawlers (anthropic-ai, ClaudeBot, …); `*` is allowed | +| Sharp (92499) | 9 | — | `Disallow: /docs/default-source/pdfs/` | +| Valley (84014) | 5 | — | 403 | +| Molina, WHA, CCHP (18126, 93689, 47579) | 5, 10, 12 | To be located | allowed | + +- **A Kaiser SBC parses cleanly** with the current parser: coverage year 2026, + 25 sections, nothing missing. Its printed title is "Covered CA_Silver 70 HMO". +- **Blue Shield's 18 regional "Trio HMO" filings share one marketing name** per + metal, so many plan IDs can point at one document. The model already allows + that: documents are unique on (url, plan_year). + +## Decisions (user, 2026-09-25, plus the master plan) + +1. **A hand-built, committed manifest**, with a check command. The HTML + "proposal helper" is dropped: most carriers' pages don't suit it. +2. **A new registry status, `mandated_disclosure`,** which gates California only. + The 18-state crawl is unchanged; gating it too is a follow-up. +3. **Health Net stays unlinked.** Its plans keep "no link" and point to Covered + California. +4. **Manual import is allowed for refusing hosts** (earlier decision): + Anthem, Sharp, Valley, and L.A. Care. L.A. Care is manual-only, because it + names AI crawlers, which we take as its intent. +5. **Base-plan SBC only.** Answers say so for CSR and AI/AN versions. Per-variant + SBCs are a follow-up. +6. **Website Terms govern (user, 2026-09-26).** Each carrier's Terms of Use were + read after this spec was first approved. + - A carrier whose Terms ban robots or scrapers is manual-only, even where + `robots.txt` allows us: Kaiser, Blue Shield, Western Health Advantage, + Anthem and L.A. Care. Valley is manual-only too, because its host blocks + bots. + - IEHP, Molina, Balance by CCHP and Sharp are crawled. Sharp's SBC folder + isn't under its disallowed path, so decision 4's "Sharp manual" is replaced. + + The evidence is in `docs/findings/ca-sbc.md`, and the rule is ADR 0026's. + +## Design + +### 1. Registry (`src/ingestion/sources/`) + +- **New status.** `PERMISSIONS` and `APPROVED` gain `mandated_disclosure`. It is + valid only for `kind = "sbc_host"`, and its `commercial_use` must be `review`. + Validation enforces both. +- **New lookup.** `sbc_host_for(url) -> RegisteredSource | None` returns the + `sbc_host` entry whose `scope_urls` prefix matches the URL. The longest prefix + wins, and it ignores the `enabled` flag. +- **Scope URLs are `https://host/…` prefixes** (added after review), so a + prefix can never also match a look-alike host. +- **New entries in `registry.toml`,** one `sbc_` per California carrier + whose SBCs we use: + - `jurisdiction = "CA"`; + - `scope_urls` as exact path prefixes; + - `access` of `crawl` or `manual`; + - the `robots` outcome with its check date. +- **Health Net** is recorded disabled, with the reason: SBCs only behind a + search form. + +### 2. Manifest (`src/ingestion/sbc/manifests/ca-.csv`, `src/ingestion/sbc/manifest.py`) + +- **Columns:** `hios_plan_id,plan_year,sbc_url,verified_on,note`. It holds + public URLs and IDs only. +- **`load_manifest(path, year)`** checks the file on its own, with no database: + - the header; + - the ID matches `^\d{5}CA\d{7}$`; + - the year equals the file's year; + - the URL passes `fetch.unsafe_reason` and has a registered `sbc_host` entry; + - no plan appears twice. + + Any failure raises `ManifestError` naming the line. +- **`apply_manifest(session, rows, year) -> ApplyReport`** makes the manifest + authoritative for California: + - It sets `benefits_url` on every `catalog_source = 'ca_sbe_puf'` plan of the + year. A plan gets its row's URL when that row's host entry is enabled, and + NULL otherwise. + - A manifest plan that isn't loaded raises. + - It reports: linked plans, plans with no row, rows skipped because their host + is disabled, and distinct documents. + + Disabling a carrier and re-applying is the removal path. PDFs are kept (ADR 0016). +- **Where it runs:** + - `make apply-sbc-manifest YEAR=` runs it. + - `make ingest-ca-plans` also runs it after a load when the year's manifest + exists. If the manifest has an error, the plans stay loaded, and the command + says the links weren't applied and exits non-zero. +- **`make check-sbc-manifest YEAR=`** only reads. After the ingest it lists: + - each California document whose printed title doesn't contain every word of + the plan's marketing name, for a person to judge; + - rows still unread, split into awaiting manual import and failed with status; + - plans with no link. + + This is the "verify" step. robots, PDF, parse and coverage year are already + checked by the ingest. + +### 3. Crawl (`ingest.py`, `report.py`, `plans.resolve_states`) + +- **California becomes a valid state.** `resolve_states(raw, allowed=MARKETPLACE_STATES)` + gains a parameter. The SBC ingest and report pass `CATALOG_STATES`, so + `make ingest-sbc STATES=CA` works; `ALL` stays the 30 API states. +- **California URLs are gated.** For a plan in `FILED_RATE_STATES`, the URL is + fetched only when its host entry is enabled with `access = "crawl"`. A + `manual` entry's documents are never requested; they are counted as "awaiting + `make import-sbc`". The check runs in `ingest.execute`, where every + California fetch starts, so a mis-applied link can't slip through. +- **Everything else is reused unchanged:** fetch (robots, pacing, redirects, + PDF check), parse, `wrong_year`, archive. + +### 4. Manual import (`src/ingestion/sbc/import_pdf.py`, `make import-sbc`) + +- **Invocation:** `make import-sbc YEAR= FILE=path URL=sbc_url`, or + `make import-sbc YEAR= DIR=folder`. `DIR` matches each file to the one + manual-access link whose URL basename equals the file name. +- **Checks, before anything is written:** + - the URL is a `benefits_url` of a plan in that year; + - its host entry is enabled; + - the file starts with `%PDF-` within 1 KB (`looks_like_pdf`, split out of `fetch._read`); + - the file is at most `MAX_BYTES`. +- **Then:** + 1. Archive any cached file with a different hash (`ingest.archive`). + 2. Copy the file to `cache_path(url, year)` with `fetch.save`. The user's file + is left where it is. + 3. Call `ingest_document(url, year, stored, acquisition="manual")`. + `fetch_pdf` serves the cached copy, so no request is made. + +### 5. Provenance (migration + model) + +- **New column.** `sbc_documents.acquisition`: String(8), NOT NULL, + server_default `'crawl'`, check `IN ('crawl','manual')`. +- **`_store` writes it only when given,** so a re-parse keeps `manual`. +- **A manual document is never re-requested:** + - `refresh-sbc` skips it and counts it. + - `ingest-sbc` re-parses it from disk after a parser bump. + - If its file is missing, the ingest reports "re-import" instead of fetching. +- **`sbc-report`** shows crawled and manual counts. `check-sbc-manifest` lists + the plans awaiting import. + +### 6. Answers + +- **A server-written notice on California answers.** When `plan_coverage` + quotes a filed-rate state's plan, the answer opens with: "Summaries of + Benefits and Coverage quoted here for Covered California plans are for each + plan's standard version; people who qualify for cost-sharing reductions or + American Indian and Alaska Native cost sharing pay less than they show." +- **It is written by the server, not left to the model.** This is the same + lesson as the prior-year notice. The implementation first put a `note` in the + tool result and a prompt line asking the model to say it; in a live question + the model left it out, so it now uses `ToolOutcome.notice`. +- **API-state plans are unchanged.** What their `benefits_url` points to hasn't + been checked, so nothing is claimed about it. + +### 7. Docs + +- **ADR 0026:** the manifest, manual import, `mandated_disclosure`, California + gating. It amends: + - ADR 0013: a person downloading in a browser is not a crawler working around + a refusal, and a named AI block is taken as intent; + - ADR 0019: manual documents are not refreshed; + - ADR 0025: the new status. +- **Findings:** `docs/findings/ca-sbc.md`, with the per-carrier research, what + was checked, and the counts. +- **Runbooks:** `docs/runbooks/california.md` gets a yearly SBC section; + `docs/runbooks/sbc.md` gets import and awaiting-import notes. +- **README and CHANGELOG.** + +## Data work (after the code) + +1. Locate each carrier's federal SBCs, and record the evidence in the findings + file: + - by crawl: IEHP, Molina, Balance by CCHP, Sharp; + - manual only: Kaiser, Blue Shield, WHA, Anthem, L.A. Care, Valley (decision 6). +2. Write `ca-2026.csv`, then run `apply-sbc-manifest`, `ingest-sbc STATES=CA + YEAR=2026` and `check-sbc-manifest`. +3. **Your step:** download the manual carriers' SBCs in a browser, then run + `make import-sbc DIR=… YEAR=2026`. +4. Run `sbc-report STATES=CA VERIFY=1`. Then a real question, e.g. Kaiser Silver + 70 HMO and urgent care, cited. Then `scripts/eval_sbc_ranking.py`, which must + stay at or above 97%. + +## Out of scope + +- The Health Net links. +- Per-variant SBCs. +- Gating the 18-state crawl by the registry. +- SBC embeddings (SP3). +- Spanish documents. + +## Tests (written after the features, as you direct) + +`test_registry` (new status rules, `sbc_host_for`), `test_sbc_manifest`, +`test_sbc_import`, additions to `test_sbc_ingest`, `test_sbc_refresh`, +`test_sbc_report`, `test_ca_puf_load` (auto-apply), `test_tools`, +`test_generation` (note), and `test_sbc_hygiene` stays green. There is no real +document in any test. diff --git a/migrations/versions/20260925_1930-3b1d6e2a9c47_add_sbc_document_acquisition.py b/migrations/versions/20260925_1930-3b1d6e2a9c47_add_sbc_document_acquisition.py new file mode 100644 index 0000000..1d0da99 --- /dev/null +++ b/migrations/versions/20260925_1930-3b1d6e2a9c47_add_sbc_document_acquisition.py @@ -0,0 +1,31 @@ +"""add sbc_documents.acquisition + +Revision ID: 3b1d6e2a9c47 +Revises: 15bb18a57fac +Create Date: 2026-09-25 19:30:00.000000+00:00 + +""" + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op + +# revision identifiers, used by Alembic. +revision: str = "3b1d6e2a9c47" +down_revision: str | Sequence[str] | None = "15bb18a57fac" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + + +def upgrade() -> None: + # Every document stored so far was fetched by the crawler (ADR 0026). + op.add_column("sbc_documents", sa.Column("acquisition", sa.String(length=8), server_default="crawl", + nullable=False)) + op.create_check_constraint("ck_sbc_documents_acquisition", "sbc_documents", + "acquisition IN ('crawl', 'manual')") + + +def downgrade() -> None: + op.drop_constraint("ck_sbc_documents_acquisition", "sbc_documents", type_="check") + op.drop_column("sbc_documents", "acquisition") diff --git a/src/ingestion/ca_puf/__main__.py b/src/ingestion/ca_puf/__main__.py index d081b96..024fde5 100644 --- a/src/ingestion/ca_puf/__main__.py +++ b/src/ingestion/ca_puf/__main__.py @@ -1,5 +1,8 @@ """`make ingest-ca-plans YEAR=2026 [ZIP=path] [REFRESH=1]`: load California's plans from the CMS PUF. +Then applies the year's SBC manifest, if there is one (ADR 0026): the PUF has +no SBC links, so without it every plan is shown with none. + uv run python -m src.ingestion.ca_puf --year 2026 """ import argparse @@ -16,6 +19,7 @@ from ..marketplace_api import county_zips from ..plans import _write_zip_counties +from ..sbc.manifest import ManifestError, apply_for_year, manifest_path, print_apply from ..sources.registry import SourceNotApprovedError, require_enabled from .download import PUF_URL, NotPublishedError, download, sha256_of from .load import STATE, LoadError, load @@ -69,6 +73,19 @@ def main(args=sys.argv[1:]): if report.unrated_zips: print(f"Los Angeles ZIPs with no CMS rating area, shown unpriced: {', '.join(report.unrated_zips)}") logger.info("california plans loaded: %d plans, %d rates, file %s", report.plans, report.rates, puf.label) + _apply_sbc_links(parsed.year) + + +def _apply_sbc_links(year: int) -> None: + """The plans stay loaded whatever happens here; only their SBC links depend on it.""" + try: + applied = apply_for_year(year) + except ManifestError as exc: + sys.exit(f"Plans loaded, but their SBC links were not applied: {exc}") + if applied is None: + print(f"No SBC manifest for {year} ({manifest_path(year).name}): every plan is shown with no SBC link") + return + print_apply(applied, year) if __name__ == "__main__": diff --git a/src/ingestion/plans.py b/src/ingestion/plans.py index 29b5b9d..0d3b046 100644 --- a/src/ingestion/plans.py +++ b/src/ingestion/plans.py @@ -14,6 +14,7 @@ import argparse import sys from collections import Counter +from collections.abc import Sequence from datetime import UTC, datetime from decimal import Decimal @@ -40,19 +41,24 @@ _REQUIRED_ISSUER_FIELDS = ("id", "name", "state") -def resolve_states(raw: str) -> list[str]: - """`"tx, FL"` → `["TX", "FL"]`; `"ALL"` → every marketplace state.""" +def resolve_states(raw: str, allowed: Sequence[str] = MARKETPLACE_STATES) -> list[str]: + """`"tx, FL"` → `["TX", "FL"]`; `"ALL"` → every marketplace state. + + `allowed` widens what may be named: the SBC commands pass CATALOG_STATES, + so California's plans, loaded from the PUF, can be read too. `ALL` stays + the marketplace states, so it never starts a California crawl unasked. + """ requested = [code.strip().upper() for code in raw.split(",") if code.strip()] if not requested: raise ValueError("no states given") if "ALL" in requested: return list(MARKETPLACE_STATES) - unknown = sorted(set(requested) - set(MARKETPLACE_STATES)) + unknown = sorted(set(requested) - set(allowed)) if unknown: hint = "; California is loaded by `make ingest-ca-plans`" if "CA" in unknown else "" raise ValueError( - f"not HealthCare.gov marketplace states: {', '.join(unknown)} " + f"not states PolicyPal has plans for here: {', '.join(unknown)} " f"(they run their own exchange, or are not state codes){hint}" ) return list(dict.fromkeys(requested)) diff --git a/src/ingestion/sbc/fetch.py b/src/ingestion/sbc/fetch.py index 9db8cb1..28f219d 100644 --- a/src/ingestion/sbc/fetch.py +++ b/src/ingestion/sbc/fetch.py @@ -87,7 +87,7 @@ def _fetch(url: str, target: Path | None, validators: tuple[str | None, str | No def _download(url: str, target: Path | None, validators: tuple[str | None, str | None]) -> FetchResult: for _ in range(_MAX_REDIRECTS + 1): - refusal = _unsafe(url) or _robots_refusal(url) + refusal = unsafe_reason(url) or _robots_refusal(url) if refusal: return FetchResult("blocked", detail=refusal) @@ -126,7 +126,7 @@ def _read(response: requests.Response, target: Path | None) -> FetchResult: body += block if len(body) > MAX_BYTES: return FetchResult("too_large", detail=f"over {MAX_BYTES // (1024 * 1024)} MB") - if b"%PDF-" not in body[:_PDF_MAGIC_WINDOW]: + if not looks_like_pdf(body): return FetchResult("not_pdf", detail="response is not a PDF") served = {"etag": response.headers.get("ETag"), "last_modified": response.headers.get("Last-Modified")} @@ -136,6 +136,11 @@ def _read(response: requests.Response, target: Path | None) -> FetchResult: return FetchResult("ok", target, **served) +def looks_like_pdf(head: bytes | bytearray) -> bool: + """Whether a file's first bytes carry a PDF header, where readers look for one.""" + return b"%PDF-" in head[:_PDF_MAGIC_WINDOW] + + def save(body: bytes, target: Path) -> None: """Write a downloaded PDF into the cache, atomically. @@ -148,8 +153,8 @@ def save(body: bytes, target: Path) -> None: partial.replace(target) -def _unsafe(url: str) -> str | None: - """Why `url` must not be fetched, or None.""" +def unsafe_reason(url: str) -> str | None: + """Why `url` must not be fetched, or None. Also vets a manifest's links before they are stored.""" parts = urlsplit(url) host = (parts.hostname or "").lower() if parts.scheme != "https": diff --git a/src/ingestion/sbc/import_pdf.py b/src/ingestion/sbc/import_pdf.py new file mode 100644 index 0000000..0fe61d1 --- /dev/null +++ b/src/ingestion/sbc/import_pdf.py @@ -0,0 +1,149 @@ +"""`make import-sbc`: read an SBC that a person downloaded in a browser (ADR 0026). + +Some carriers refuse automated requests (robots.txt, a 403) or name AI +crawlers as unwelcome. The crawler never works around that. A person may still +open the carrier's page and download the plan's SBC, as any shopper would, and +this reads that file: it is copied into the cache under the plan's link, and +parsed exactly as a crawled one, with no request made. It is recorded as +`manual`, so no ingest or refresh ever asks the carrier for it. + + uv run python -m src.ingestion.sbc.import_pdf --year 2026 --file ~/Downloads/x.pdf --url https://…/x.pdf + uv run python -m src.ingestion.sbc.import_pdf --year 2026 --dir ~/Downloads/ca-sbc +""" +import argparse +import hashlib +import sys +from collections import defaultdict +from pathlib import Path, PurePosixPath +from urllib.parse import parse_qsl, unquote, urlsplit + +from sqlalchemy import select + +from src.core.db import get_session +from src.core.logging import get_logger +from src.models.plan import Plan +from src.models.sbc import SbcDocument + +from ..sources.registry import host_access +from .fetch import MAX_BYTES, cache_path, looks_like_pdf, save +from .ingest import archive, ingest_document + +logger = get_logger(__name__) + + +class SbcImportError(ValueError): + """The file or link can't be imported; nothing was written. The message says why.""" + + +def _linked(year: int) -> set[str]: + with get_session() as session: + return set(session.scalars( + select(Plan.benefits_url).where(Plan.plan_year == year, Plan.benefits_url.is_not(None)).distinct() + )) + + +def import_file(path: Path, url: str, year: int, linked: set[str] | None = None) -> str: + """Import one downloaded PDF as the document behind `url`. Returns its status.""" + if url not in (linked if linked is not None else _linked(year)): + raise SbcImportError(f"no {year} plan links to {url}; apply the SBC manifest first") + if host_access(url) is None: + raise SbcImportError(f"{url}: its host has no enabled sbc_host entry in the source registry") + if not path.is_file(): + raise SbcImportError(f"{path} is not a file") + if path.stat().st_size > MAX_BYTES: + raise SbcImportError(f"{path.name} is over {MAX_BYTES // (1024 * 1024)} MB; not a Summary of Benefits") + body = path.read_bytes() + if not looks_like_pdf(body): + raise SbcImportError(f"{path.name} is not a PDF") + + target = cache_path(url, year) + if target.exists() and hashlib.sha256(target.read_bytes()).hexdigest() != hashlib.sha256(body).hexdigest(): + archive(url, year) # the file it replaces is kept (ADR 0016) + save(body, target) + with get_session() as session: + stored = session.execute( + select(SbcDocument.status, SbcDocument.sha256) + .where(SbcDocument.url == url, SbcDocument.plan_year == year) + ).first() + status = ingest_document(url, year, stored, acquisition="manual") + logger.info("sbc imported by hand: %s -> %s", target.name, status) + return status + + +def _file_names(url: str) -> set[str]: + """The names a browser may save this link's PDF under: the path's last + segment, and any query value naming a PDF (e.g. `?fileName=x.pdf`).""" + parts = urlsplit(url) + names = {PurePosixPath(unquote(parts.path)).name} + names.update(PurePosixPath(value).name for _, value in parse_qsl(parts.query) + if value.lower().endswith(".pdf")) + return names + + +def import_folder(folder: Path, year: int) -> dict[str, str]: + """Import every PDF in `folder` whose name is the file name of exactly one manual-only link. + + Returns each file's outcome: a status, or why it was not imported. + """ + linked = _linked(year) + by_name: dict[str, list[str]] = defaultdict(list) + for url in linked: + if host_access(url) == "manual": + for name in _file_names(url): + by_name[name].append(url) + outcomes = {} + for path in sorted(folder.glob("*.pdf")): + urls = by_name.get(path.name, []) + if len(urls) != 1: + outcomes[path.name] = ("not imported: no manual-only link ends in this file name" if not urls + else f"not imported: {len(urls)} links end in this file name; use FILE= and URL=") + continue + try: + outcomes[path.name] = import_file(path, urls[0], year, linked) + except SbcImportError as exc: + outcomes[path.name] = f"not imported: {exc}" + return outcomes + + +def _path(raw: str) -> Path: + """A path as typed; `~` is expanded here because make passes it quoted.""" + return Path(raw).expanduser() + + +def parse_args(args): + parser = argparse.ArgumentParser(description="Import SBC PDFs downloaded by hand.") + parser.add_argument("--year", type=int, required=True, help="Plan year, e.g. 2026") + source = parser.add_mutually_exclusive_group(required=True) + source.add_argument("--file", type=_path, help="One downloaded PDF; needs --url") + source.add_argument("--dir", type=_path, help="A folder of downloaded PDFs, matched to links by file name") + parser.add_argument("--url", help="The plan's SBC link (from the manifest) that --file is the document behind") + parsed = parser.parse_args(args) + if parsed.file and not parsed.url: + parser.error("--file needs --url") + if parsed.dir and parsed.url: + parser.error("--url goes with --file, not --dir") + return parsed + + +def main(args=sys.argv[1:]) -> None: + parsed = parse_args(args) + if parsed.file: + try: + print(f"{parsed.file.name}: {import_file(parsed.file, parsed.url, parsed.year)}") + except SbcImportError as exc: + sys.exit(f"Not imported: {exc}") + return + if not parsed.dir.is_dir(): + sys.exit(f"{parsed.dir} is not a folder") + outcomes = import_folder(parsed.dir, parsed.year) + if not outcomes: + sys.exit(f"No PDFs in {parsed.dir}") + for name, outcome in outcomes.items(): + print(f"{name}: {outcome}") + + +if __name__ == "__main__": + from src.core.logging import setup_logging + + setup_logging() + main() diff --git a/src/ingestion/sbc/ingest.py b/src/ingestion/sbc/ingest.py index 40d14c7..8e0ece7 100644 --- a/src/ingestion/sbc/ingest.py +++ b/src/ingestion/sbc/ingest.py @@ -13,6 +13,11 @@ Its chunks go to `sbc_chunks`, apart from the corpus, which `make ingest` rebuilds without touching them. +California's links come from a hand-built manifest (ADR 0026), and each is +read only as its host's source-registry entry allows: `crawl` is fetched here, +`manual` never is — a person downloads it and `make import-sbc` reads it. A +manually imported document is re-parsed from disk, never requested again. + uv run python -m src.ingestion.sbc --states NH,DE --limit 5 """ import argparse @@ -29,6 +34,7 @@ from tqdm import tqdm from src.core.db import get_session +from src.core.exchanges import CATALOG_STATES, FILED_RATE_STATES from src.core.logging import get_logger from src.core.text import count_tokens from src.models.plan import Issuer, Plan @@ -37,8 +43,9 @@ from src.services.sbc_status import READ_STATUSES from ..chunking import make_recursive_splitter -from ..constants import SBC_ARCHIVE, SBC_REJECTED +from ..constants import SBC_ARCHIVE, SBC_RAW, SBC_REJECTED from ..plans import resolve_states, upsert +from ..sources.registry import host_access from .extract import ( PARSER_VERSION, SbcParse, @@ -73,6 +80,7 @@ class PlanRef: hios_plan_id: str name: str issuer: str + state: str def documents_for(session, states: list[str], year: int, @@ -82,7 +90,7 @@ def documents_for(session, states: list[str], year: int, `issuer_ids` narrows it to those HIOS issuers' plans. """ stmt = ( - select(Plan.benefits_url, Plan.hios_plan_id, Plan.marketing_name, Issuer.name) + select(Plan.benefits_url, Plan.hios_plan_id, Plan.marketing_name, Issuer.name, Plan.state) .join(Issuer, Issuer.id == Plan.issuer_id) .where(Plan.plan_year == year, Plan.state.in_(states), Plan.benefits_url.is_not(None)) .order_by(Issuer.name, Plan.hios_plan_id) @@ -91,8 +99,8 @@ def documents_for(session, states: list[str], year: int, stmt = stmt.where(Issuer.hios_issuer_id.in_(issuer_ids)) rows = session.execute(stmt).all() documents = defaultdict(list) - for url, plan_id, name, issuer in rows: - documents[url].append(PlanRef(plan_id, name, issuer)) + for url, plan_id, name, issuer, state in rows: + documents[url].append(PlanRef(plan_id, name, issuer, state)) return dict(documents) @@ -117,12 +125,17 @@ def build_chunks(url: str, year: int, parse: SbcParse) -> list[dict]: return chunks -def ingest_document(url: str, year: int, stored=None) -> str: - """Fetch, parse and store one SBC. Returns its status. The PDF is always kept.""" +def ingest_document(url: str, year: int, stored=None, acquisition: str | None = None) -> str: + """Fetch, parse and store one SBC. Returns its status. The PDF is always kept. + + A file already in the cache is read from there, with no request, which is + how a manual import is parsed. `acquisition` is recorded only when given, + so re-parsing an imported document keeps it marked `manual`. + """ _restore_rejected(url, year, stored) fetched = fetch_pdf(url, year) if fetched.status != "ok": - _store(url, year, fetched) + _store(url, year, fetched, acquisition=acquisition) return fetched.status digest = hashlib.sha256(fetched.path.read_bytes()).hexdigest() @@ -131,13 +144,13 @@ def ingest_document(url: str, year: int, stored=None) -> str: try: pages, parse = read_parsed(fetched.path) except SbcParseError as exc: - return _reject(url, year, "unparseable", str(exc), digest, downloaded) + return _reject(url, year, "unparseable", str(exc), digest, downloaded, acquisition) if parse.coverage_year != year: status = _reject(url, year, "wrong_year", f"coverage period starts in {parse.coverage_year}", - digest, downloaded) + digest, downloaded, acquisition) # Moved aside, not deleted: the issuer may correct the file at the # same URL, and only a fresh download would see it. - rejected = SBC_REJECTED / str(year) / f"{fetched.path.stem}-{digest[:12]}.pdf" + rejected = _rejected_path(url, year, digest) rejected.parent.mkdir(parents=True, exist_ok=True) fetched.path.replace(rejected) return status @@ -148,10 +161,11 @@ def ingest_document(url: str, year: int, stored=None) -> str: read = replace(fetched, status="partial", detail=f"missing {missing}") if missing else fetched try: _store(url, year, read, sha256=digest, pages=len(pages), title=parse.title, - chunks=build_chunks(url, year, parse), parser_version=PARSER_VERSION, fetched_at=downloaded) + chunks=build_chunks(url, year, parse), parser_version=PARSER_VERSION, fetched_at=downloaded, + acquisition=acquisition) except DataError: # One document the database refuses must not end a run over thousands. - return _reject(url, year, "unparseable", "text the database refuses", digest, downloaded) + return _reject(url, year, "unparseable", "text the database refuses", digest, downloaded, acquisition) return read.status @@ -165,35 +179,53 @@ def _restore_rejected(url: str, year: int, stored) -> None: if stored is None or stored.status != "wrong_year" or not stored.sha256: return current = cache_path(url, year) - rejected = SBC_REJECTED / str(year) / f"{current.stem}-{stored.sha256[:12]}.pdf" + rejected = _rejected_path(url, year, stored.sha256) if rejected.exists() and not current.exists(): current.parent.mkdir(parents=True, exist_ok=True) rejected.replace(current) -def _reject(url: str, year: int, status: str, detail: str, sha256: str, downloaded: datetime) -> str: +def _rejected_path(url: str, year: int, sha256: str): + """Where a `wrong_year` file is kept aside (ADR 0018).""" + return SBC_REJECTED / str(year) / f"{cache_path(url, year).stem}-{sha256[:12]}.pdf" + + +def _on_disk(url: str, year: int, stored) -> bool: + """Whether the document's PDF is kept here, in the cache or set aside as the wrong year.""" + return cache_path(url, year).exists() or bool( + stored is not None and stored.status == "wrong_year" and stored.sha256 + and _rejected_path(url, year, stored.sha256).exists()) + + +def _reject(url: str, year: int, status: str, detail: str, sha256: str, downloaded: datetime, + acquisition: str | None = None) -> str: """A file the parser turned down. Which file, and which parser, are kept (ADR 0018).""" _store(url, year, FetchResult(status, detail=detail), sha256=sha256, parser_version=PARSER_VERSION, - fetched_at=downloaded) + fetched_at=downloaded, acquisition=acquisition) return status def _store(url: str, year: int, fetched: FetchResult, *, sha256: str | None = None, pages: int | None = None, title: str | None = None, chunks: list[dict] | None = None, - parser_version: int | None = None, fetched_at: datetime | None = None) -> None: + parser_version: int | None = None, fetched_at: datetime | None = None, + acquisition: str | None = None) -> None: """Record the attempt and replace the document's chunks, in one transaction. A document that fails now loses the chunks an earlier run gave it: an answer must never quote an SBC that is no longer the plan's. Only a file the parser judged keeps its parser version, so a parser change reads it again (ADR 0018). A fetch that failed is stamped with the attempt's time. + `acquisition` is written only when given, so it survives a re-parse. """ + row = { + "url": url, "plan_year": year, "status": fetched.status, "detail": fetched.detail, + "sha256": sha256, "pages": pages, "title": title, "fetched_at": fetched_at or datetime.now(UTC), + "parser_version": parser_version, + } + if acquisition is not None: + row["acquisition"] = acquisition with get_session() as session: - upsert(session, SbcDocument, [{ - "url": url, "plan_year": year, "status": fetched.status, "detail": fetched.detail, - "sha256": sha256, "pages": pages, "title": title, "fetched_at": fetched_at or datetime.now(UTC), - "parser_version": parser_version, - }], "uq_sbc_documents_url_plan_year", ("url", "plan_year")) + upsert(session, SbcDocument, [row], "uq_sbc_documents_url_plan_year", ("url", "plan_year")) document_id = session.scalar( select(SbcDocument.id).where(SbcDocument.url == url, SbcDocument.plan_year == year) ) @@ -217,7 +249,7 @@ def refresh_document(url: str, year: int, stored) -> str: # its text stays, and the next refresh asks again. return "unreachable" if fetched.status != "ok": - _archive(url, year) + archive(url, year) _store(url, year, fetched) return fetched.status @@ -225,14 +257,14 @@ def refresh_document(url: str, year: int, stored) -> str: _checked(url, year, fetched) return "unchanged" - _archive(url, year, stored.sha256) + archive(url, year, stored.sha256) save(fetched.body, cache_path(url, year)) status = ingest_document(url, year) _checked(url, year, fetched) return f"changed:{status}" -def _archive(url: str, year: int, sha256: str | None = None) -> None: +def archive(url: str, year: int, sha256: str | None = None) -> None: """Move the file a new one replaces out of the cache, keeping it for good (ADR 0016).""" current = cache_path(url, year) if not current.exists(): @@ -268,6 +300,17 @@ def _needs_reading(url: str, row, year: int, refresh: bool) -> bool: or (row.status in READ_STATUSES and not cache_path(url, year).exists())) +def _access(url: str, plans: list[PlanRef]) -> str | None: + """How this run may read a link: "crawl", "manual", or None when the registry doesn't allow it. + + Only a filed-rate state's links are gated (ADR 0026); the HealthCare.gov + states' links come from CMS and are read as before. + """ + if not any(plan.state in FILED_RATE_STATES for plan in plans): + return "crawl" + return host_access(url) + + def execute(states: list[str], year: int, limit: int | None = None, issuer_ids: Collection[str] | None = None, refresh: bool = False) -> Counter: """Read what is new or outdated; with `refresh`, ask about every stored document too. @@ -278,26 +321,50 @@ def execute(states: list[str], year: int, limit: int | None = None, documents = documents_for(session, states, year, issuer_ids) stored = {row.url: row for row in session.execute( select(SbcDocument.url, SbcDocument.status, SbcDocument.parser_version, SbcDocument.sha256, - SbcDocument.etag, SbcDocument.last_modified).where(SbcDocument.plan_year == year) + SbcDocument.etag, SbcDocument.last_modified, SbcDocument.acquisition) + .where(SbcDocument.plan_year == year) ).all()} if not documents: - sys.exit(f"No catalog plans with an SBC link for {', '.join(states)} in {year}. " - f"Run `make ingest-plans STATES={','.join(states)}` first.") - pending = [url for url in documents if _needs_reading(url, stored.get(url), year, refresh)] + first = (f"`make apply-sbc-manifest YEAR={year}`" if set(states) <= set(FILED_RATE_STATES) + else f"`make ingest-plans STATES={','.join(states)}`") + sys.exit(f"No catalog plans with an SBC link for {', '.join(states)} in {year}. Run {first} first.") + pending, awaiting, reimport, unapproved = [], [], [], [] + for url, plans in documents.items(): + row = stored.get(url) + if row is not None and row.acquisition == "manual": + # A person's download is never requested again, not even by a + # refresh: it is only re-parsed from disk (ADR 0026). + if _needs_reading(url, row, year, refresh=False): + (pending if _on_disk(url, year, row) else reimport).append(url) + continue + if not _needs_reading(url, row, year, refresh): + continue + access = _access(url, plans) + {"crawl": pending, "manual": awaiting}.get(access, unapproved).append(url) urls = pending[:limit] + held = {*pending, *awaiting, *reimport, *unapproved} skipped = sum(1 for url in documents - if url not in pending and stored.get(url) and stored[url].status not in READ_STATUSES) + if url not in held and stored.get(url) and stored[url].status not in READ_STATUSES) print(f"{len(documents)} SBC documents behind the {', '.join(states)} plans for {year}; " - f"{len(documents) - len(pending)} already current, " + f"{len(documents) - len(held) - skipped} already current, " f"{'re-checking' if refresh else 'reading'} {len(urls)}") if skipped: print(f"{skipped} recorded failures skipped; `make refresh-sbc STATES={','.join(states)}` retries them") + if awaiting: + print(f"{len(awaiting)} awaiting a manual import (their host is manual-only): " + f"`make import-sbc YEAR={year} DIR=`") + if reimport: + print(f"{len(reimport)} manually imported PDFs are missing from {SBC_RAW / str(year)}: " + f"import them again") + if unapproved: + print(f"{len(unapproved)} links skipped: their host has no enabled sbc_host entry in the source registry") statuses = Counter() failures = defaultdict(list) # (issuer, status) -> plans for url in tqdm(urls, desc="SBCs"): row = stored.get(url) - status = refresh_document(url, year, row) if refresh and row and row.status in READ_STATUSES \ + status = refresh_document(url, year, row) \ + if refresh and row and row.status in READ_STATUSES and row.acquisition != "manual" \ else ingest_document(url, year, row) statuses[status] += 1 if status not in KEPT_OUTCOMES: @@ -319,7 +386,8 @@ def execute(states: list[str], year: int, limit: int | None = None, def parse_args(args): parser = argparse.ArgumentParser(description="Ingest Summary of Benefits and Coverage PDFs for catalog plans.") - parser.add_argument("--states", required=True, help="Comma-separated state codes, or ALL, e.g. NH,DE") + parser.add_argument("--states", required=True, + help="Comma-separated state codes, or ALL (the HealthCare.gov states), e.g. NH,DE or CA") parser.add_argument("--year", type=int, default=datetime.now(UTC).year, help="Plan year (default: this year); must match an ingested catalog year") parser.add_argument("--limit", type=int, default=None, help="Read at most this many documents, for a smoke run") @@ -331,7 +399,7 @@ def parse_args(args): issuers.add_argument("--issuers", help="Only these HIOS issuer IDs' plans, comma-separated, e.g. 40788,66252") parsed = parser.parse_args(args) try: - parsed.states = resolve_states(parsed.states) + parsed.states = resolve_states(parsed.states, CATALOG_STATES) except ValueError as exc: parser.error(str(exc)) if parsed.limit is not None and parsed.limit < 1: diff --git a/src/ingestion/sbc/manifest.py b/src/ingestion/sbc/manifest.py new file mode 100644 index 0000000..b42cc96 --- /dev/null +++ b/src/ingestion/sbc/manifest.py @@ -0,0 +1,240 @@ +"""California's SBC links: a committed, hand-built list, and the commands around it (ADR 0026). + +CMS's California PUF has no SBC links, so every California plan's link comes +from `manifests/ca-.csv`: which carrier document is which plan's, found +and checked by a person. Each row's host must have an `sbc_host` entry in the +source registry. Disabling that entry and applying again unlinks its plans, +which is how a carrier is removed. + + uv run python -m src.ingestion.sbc.manifest apply --year 2026 + uv run python -m src.ingestion.sbc.manifest check --year 2026 +""" +import argparse +import csv +import re +import sys +from collections import Counter +from dataclasses import dataclass, field +from datetime import date +from pathlib import Path + +from sqlalchemy import select, update + +from src.core.db import get_session +from src.core.logging import get_logger +from src.models.plan import Issuer, Plan +from src.models.sbc import SbcDocument +from src.services.sbc_status import READ_STATUSES, plan_sbc_status, sbc_document_join + +from ..ca_puf.load import CATALOG_SOURCE +from ..sources.registry import host_access, sbc_host_for +from .fetch import unsafe_reason + +logger = get_logger(__name__) + +MANIFEST_DIR = Path(__file__).with_name("manifests") +COLUMNS = ("hios_plan_id", "plan_year", "sbc_url", "verified_on", "note") +_PLAN_ID = re.compile(r"^\d{5}CA\d{7}$") +_WORD = re.compile(r"[a-z0-9]+") + + +class ManifestError(ValueError): + """The manifest breaks a rule; nothing was applied. The message names the line.""" + + +@dataclass(frozen=True) +class ManifestRow: + hios_plan_id: str + sbc_url: str + verified_on: date + note: str + + +@dataclass(frozen=True) +class ApplyReport: + linked: int # plans given a link + documents: int # distinct links among them + unlisted: tuple[str, ...] # loaded plans with no row: no link + disabled: dict[str, int] # registry entry -> plans left unlinked because it is disabled + + +def manifest_path(year: int) -> Path: + return MANIFEST_DIR / f"ca-{year}.csv" + + +def load_manifest(path: Path, year: int) -> tuple[ManifestRow, ...]: + """The manifest's rows, every one checked; raises ManifestError on the first bad line.""" + try: + with open(path, newline="", encoding="utf-8") as handle: + reader = csv.DictReader(handle) + if tuple(reader.fieldnames or ()) != COLUMNS: + raise ManifestError(f"{path.name}: the header must be {','.join(COLUMNS)}") + rows = [_row(raw, reader.line_num, year) for raw in reader] + except (OSError, UnicodeDecodeError) as exc: + raise ManifestError(f"{path.name} could not be read: {exc}") from None + seen = Counter(row.hios_plan_id for row in rows) + if twice := sorted(plan for plan, count in seen.items() if count > 1): + raise ManifestError(f"{path.name}: plan {twice[0]} is listed more than once") + return tuple(rows) + + +def _row(raw: dict, line: int, year: int) -> ManifestRow: + where = f"line {line}" + if None in raw or any(value is None for value in raw.values()): + raise ManifestError(f"{where}: expected {len(COLUMNS)} columns") + plan_id, url = raw["hios_plan_id"].strip(), raw["sbc_url"].strip() + if not _PLAN_ID.match(plan_id): + raise ManifestError(f"{where}: {plan_id!r} is not a California base plan ID (e.g. 40513CA0380003)") + if raw["plan_year"].strip() != str(year): + raise ManifestError(f"{where}: plan year {raw['plan_year']!r} is not {year}") + if reason := unsafe_reason(url): + raise ManifestError(f"{where}: the link is {reason}") + entry = sbc_host_for(url) + if entry is None or entry.jurisdiction != "CA": + raise ManifestError(f"{where}: no California sbc_host entry in the source registry covers {url}") + try: + verified = date.fromisoformat(raw["verified_on"].strip()) + except ValueError: + raise ManifestError(f"{where}: verified_on must be a date, e.g. 2026-09-25") from None + return ManifestRow(plan_id, url, verified, raw["note"].strip()) + + +def apply_manifest(session, rows: tuple[ManifestRow, ...], year: int) -> ApplyReport: + """Set every California plan's link for the year from the manifest; the caller commits. + + Authoritative: a plan with no row, or whose row's host is disabled, is left + with no link, so a removed row or carrier stops being read. + """ + plans = dict(session.execute( + select(Plan.hios_plan_id, Plan.id).where(Plan.plan_year == year, Plan.catalog_source == CATALOG_SOURCE) + ).all()) + if unknown := sorted({row.hios_plan_id for row in rows} - set(plans)): + raise ManifestError(f"plan {unknown[0]} is in the manifest but not among the {year} California plans") + + by_plan = {row.hios_plan_id: row for row in rows} + links: dict[str, str | None] = {} + disabled: Counter = Counter() + for plan_id in plans: + row = by_plan.get(plan_id) + if row is not None and host_access(row.sbc_url) is None: + disabled[sbc_host_for(row.sbc_url).id] += 1 + row = None + links[plan_id] = row.sbc_url if row else None + + if plans: + session.execute(update(Plan), [{"id": plans[plan_id], "benefits_url": url} for plan_id, url in links.items()]) + linked = {url for url in links.values() if url} + return ApplyReport(linked=sum(1 for url in links.values() if url), documents=len(linked), + unlisted=tuple(sorted(plan for plan in plans if plan not in by_plan)), disabled=dict(disabled)) + + +def apply_for_year(year: int) -> ApplyReport | None: + """Load and apply the year's manifest in its own transaction; None when there is none.""" + path = manifest_path(year) + if not path.exists(): + return None + rows = load_manifest(path, year) + with get_session() as session: + return apply_manifest(session, rows, year) + + +def print_apply(report: ApplyReport, year: int) -> None: + print(f"{report.linked} California plans for {year} linked to {report.documents} SBC documents; " + f"{len(report.unlisted)} not in the manifest, so shown with no link") + for source, count in sorted(report.disabled.items()): + print(f" {count} plans unlinked: registry entry {source} is disabled") + logger.info("sbc manifest applied for %d: %d plans, %d documents", year, report.linked, report.documents) + + +def name_matches(marketing_name: str, printed_title: str | None) -> bool: + """Whether every word of the plan's name is printed in the SBC's title. + + A cheap guard against a row pointing at the wrong plan's document; a person + judges each mismatch, since carriers print names their own way. + """ + return bool(printed_title) and set(_WORD.findall(marketing_name.lower())) <= set( + _WORD.findall(printed_title.lower())) + + +@dataclass +class CheckReport: + """What a person should look at after an ingest. Each list holds (issuer, plan ID, name, …) rows.""" + read: int = 0 + manual: int = 0 # of `read`, how many came from a manual import + name_mismatch: list[tuple] = field(default_factory=list) + failed: list[tuple] = field(default_factory=list) + awaiting_import: list[tuple] = field(default_factory=list) + not_read: list[tuple] = field(default_factory=list) + no_link: list[tuple] = field(default_factory=list) + + +def check(session, year: int) -> CheckReport: + """Every California plan of the year, sorted into what needs a person's eye; reads only.""" + rows = session.execute( + select(Plan.hios_plan_id, Plan.marketing_name, Plan.benefits_url, Issuer.name, plan_sbc_status(), + SbcDocument.title, SbcDocument.acquisition) + .join(Issuer, Issuer.id == Plan.issuer_id) + .outerjoin(SbcDocument, sbc_document_join()) + .where(Plan.plan_year == year, Plan.catalog_source == CATALOG_SOURCE) + .order_by(Issuer.name, Plan.hios_plan_id) + ).all() + report = CheckReport() + for plan_id, name, url, issuer, status, title, acquisition in rows: + plan = (issuer, plan_id, name) + if status == "no_link": + report.no_link.append(plan) + elif status == "not_read": + (report.awaiting_import if host_access(url) == "manual" else report.not_read).append(plan) + elif status not in READ_STATUSES: + report.failed.append((*plan, status)) + else: + report.read += 1 + report.manual += acquisition == "manual" + if not name_matches(name, title): + report.name_mismatch.append((*plan, title)) + return report + + +def print_check(report: CheckReport, year: int) -> None: + print(f"California SBCs for {year}: {report.read} plans read ({report.manual} from manual imports)") + sections = ( + (report.name_mismatch, "Printed title doesn't carry the plan's name: check the row points at the right document"), + (report.failed, "Linked, but the document could not be read"), + (report.awaiting_import, "Awaiting `make import-sbc` (the carrier's documents are manual-only)"), + (report.not_read, "Linked, not read yet: run `make ingest-sbc STATES=CA`"), + (report.no_link, "No link in the manifest"), + ) + for plans, heading in sections: + if plans: + print(f"\n{heading}: {len(plans)}") + for plan in plans: + print(" " + " | ".join(str(value) for value in plan)) + + +def parse_args(args): + parser = argparse.ArgumentParser(description="Apply or check California's SBC manifest.") + parser.add_argument("command", choices=("apply", "check")) + parser.add_argument("--year", type=int, required=True, help="Plan year, e.g. 2026") + return parser.parse_args(args) + + +def main(args=sys.argv[1:]) -> None: + parsed = parse_args(args) + if parsed.command == "check": + with get_session() as session: + print_check(check(session, parsed.year), parsed.year) + return + try: + report = apply_for_year(parsed.year) + except ManifestError as exc: + sys.exit(f"SBC manifest not applied: {exc}") + if report is None: + sys.exit(f"No manifest for {parsed.year}: expected {manifest_path(parsed.year)}") + print_apply(report, parsed.year) + + +if __name__ == "__main__": + from src.core.logging import setup_logging + + setup_logging() + main() diff --git a/src/ingestion/sbc/manifests/ca-2026.csv b/src/ingestion/sbc/manifests/ca-2026.csv new file mode 100644 index 0000000..06ec876 --- /dev/null +++ b/src/ingestion/sbc/manifests/ca-2026.csv @@ -0,0 +1,115 @@ +hios_plan_id,plan_year,sbc_url,verified_on,note +40513CA0380001,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380001-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +40513CA0380002,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380002-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +40513CA0380003,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380003-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +40513CA0380004,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380004-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +40513CA0380005,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380005-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +40513CA0380006,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380006-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +40513CA0380013,2026,https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/2026/40513CA0380013-01-en-2026.pdf,2026-09-26,named by HIOS ID (-01 standard); manual only +51396CA0010001,2026,https://www.iehp.org/content/dam/iehp-org/en/documents/coveredcalifornia/summary-of-benefits-and-coverage/2026/on-exchange/2026-Minimum-Coverage-SBC.pdf,2026-09-26,"listed on IEHP's Covered California page, by metal" +51396CA0010002,2026,https://www.iehp.org/content/dam/iehp-org/en/documents/coveredcalifornia/summary-of-benefits-and-coverage/2026/on-exchange/2026-Bronze-60-SBC.pdf,2026-09-26,"listed on IEHP's Covered California page, by metal" +51396CA0010003,2026,https://www.iehp.org/content/dam/iehp-org/en/documents/coveredcalifornia/summary-of-benefits-and-coverage/2026/on-exchange/2026-Silver-70-SBC.pdf,2026-09-26,"listed on IEHP's Covered California page, by metal" +51396CA0010004,2026,https://www.iehp.org/content/dam/iehp-org/en/documents/coveredcalifornia/summary-of-benefits-and-coverage/2026/on-exchange/2026-Gold-80-SBC.pdf,2026-09-26,"listed on IEHP's Covered California page, by metal" +51396CA0010005,2026,https://www.iehp.org/content/dam/iehp-org/en/documents/coveredcalifornia/summary-of-benefits-and-coverage/2026/on-exchange/2026-Platinum-90-SBC.pdf,2026-09-26,"listed on IEHP's Covered California page, by metal" +18126CA0010001,2026,https://www.molinamarketplace.com/marketplace/ca/en-us/-/media/Molina/PublicWebsite/PDF/members/ca/en-us/Marketplace/2026/CA26SBCE_P90_1.pdf,2026-09-26,beside the listed EOC (EOCE->SBCE); printed name checked +18126CA0010002,2026,https://www.molinamarketplace.com/marketplace/ca/en-us/-/media/Molina/PublicWebsite/PDF/members/ca/en-us/Marketplace/2026/CA26SBCE_G80_1.pdf,2026-09-26,beside the listed EOC (EOCE->SBCE); printed name checked +18126CA0010003,2026,https://www.molinamarketplace.com/marketplace/ca/en-us/-/media/Molina/PublicWebsite/PDF/members/ca/en-us/Marketplace/2026/CA26SBCE_S70_1.pdf,2026-09-26,beside the listed EOC (EOCE->SBCE); printed name checked +18126CA0010004,2026,https://www.molinamarketplace.com/marketplace/ca/en-us/-/media/Molina/PublicWebsite/PDF/members/ca/en-us/Marketplace/2026/CA26SBCE_B60_1.pdf,2026-09-26,beside the listed EOC (EOCE->SBCE); printed name checked +18126CA0010005,2026,https://www.molinamarketplace.com/marketplace/ca/en-us/-/media/Molina/PublicWebsite/PDF/members/ca/en-us/Marketplace/2026/CA26SBCE_M1_1.pdf,2026-09-26,beside the listed EOC (EOCE->SBCE); printed name checked +92499CA0020001,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26750_sharp-platinum-90-hmo.pdf?sfvrsn=f323a680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020002,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26752_sharp-platinum-90-hmo.pdf?sfvrsn=ea23a680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020003,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26754_sharp-gold-80-hmo.pdf?sfvrsn=e223a680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020004,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26756_sharp-gold-80-hmo.pdf?sfvrsn=1d2ca680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020005,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26758_sharp-silver-70-hmo.pdf?sfvrsn=1c2ca680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020006,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26760_sharp-silver-70-hmo.pdf?sfvrsn=9e2ca680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020008,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26768_sharp-bronze-60-hmo.pdf?sfvrsn=992ca680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020009,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26770_sharp-bronze-60-hdhp-hmo.pdf?sfvrsn=852ca680_1,2026-09-26,prints its own HIOS ID (-01) +92499CA0020010,2026,https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/26772_sharp-minimum-coverage-hmo.pdf?sfvrsn=8a2ca680_1,2026-09-26,prints its own HIOS ID (-01) +70285CA1250001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_PPO__ML002166_01-26_SBC.pdf,2026-09-26,Covered California file on Blue Shield's IFP document page; manual only +70285CA1270001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_PPO__ML002169_01-26_SBC.pdf,2026-09-26,Covered California file on Blue Shield's IFP document page; manual only +70285CA1290001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_PPO__ML002175_01-26_SBC.pdf,2026-09-26,Covered California file on Blue Shield's IFP document page; manual only +70285CA1310001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Bronze_60_PPO__ML002187_01-26_SBC.pdf,2026-09-26,Covered California file on Blue Shield's IFP document page; manual only +70285CA1320001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Bronze_60_HDHP_PPO__ML002193_01-26_SBC.pdf,2026-09-26,Covered California file on Blue Shield's IFP document page; manual only +70285CA1350001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Minimum_Coverage_PPO__ML002196_01-26_SBC.pdf,2026-09-26,Covered California file on Blue Shield's IFP document page; manual only +70285CA8040001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 1: one SBC per metal serves all 18; manual only +70285CA8040002,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 2: one SBC per metal serves all 18; manual only +70285CA8040003,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 3: one SBC per metal serves all 18; manual only +70285CA8040004,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 4: one SBC per metal serves all 18; manual only +70285CA8040005,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 5: one SBC per metal serves all 18; manual only +70285CA8040006,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 6: one SBC per metal serves all 18; manual only +70285CA8040007,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 7: one SBC per metal serves all 18; manual only +70285CA8040008,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 8: one SBC per metal serves all 18; manual only +70285CA8040009,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 9: one SBC per metal serves all 18; manual only +70285CA8040010,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 10: one SBC per metal serves all 18; manual only +70285CA8040011,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 11: one SBC per metal serves all 18; manual only +70285CA8040012,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 12: one SBC per metal serves all 18; manual only +70285CA8040014,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 14: one SBC per metal serves all 18; manual only +70285CA8040015,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 15: one SBC per metal serves all 18; manual only +70285CA8040016,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 16: one SBC per metal serves all 18; manual only +70285CA8040017,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 17: one SBC per metal serves all 18; manual only +70285CA8040018,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 18: one SBC per metal serves all 18; manual only +70285CA8040019,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Gold_80_Trio_HMO__MI011314_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 19: one SBC per metal serves all 18; manual only +70285CA8050001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 1: one SBC per metal serves all 18; manual only +70285CA8050002,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 2: one SBC per metal serves all 18; manual only +70285CA8050003,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 3: one SBC per metal serves all 18; manual only +70285CA8050004,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 4: one SBC per metal serves all 18; manual only +70285CA8050005,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 5: one SBC per metal serves all 18; manual only +70285CA8050006,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 6: one SBC per metal serves all 18; manual only +70285CA8050007,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 7: one SBC per metal serves all 18; manual only +70285CA8050008,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 8: one SBC per metal serves all 18; manual only +70285CA8050009,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 9: one SBC per metal serves all 18; manual only +70285CA8050010,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 10: one SBC per metal serves all 18; manual only +70285CA8050011,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 11: one SBC per metal serves all 18; manual only +70285CA8050012,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 12: one SBC per metal serves all 18; manual only +70285CA8050014,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 14: one SBC per metal serves all 18; manual only +70285CA8050015,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 15: one SBC per metal serves all 18; manual only +70285CA8050016,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 16: one SBC per metal serves all 18; manual only +70285CA8050017,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 17: one SBC per metal serves all 18; manual only +70285CA8050018,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 18: one SBC per metal serves all 18; manual only +70285CA8050019,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Platinum_90_Trio_HMO__MI011317_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 19: one SBC per metal serves all 18; manual only +70285CA8060001,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 1: one SBC per metal serves all 18; manual only +70285CA8060002,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 2: one SBC per metal serves all 18; manual only +70285CA8060003,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 3: one SBC per metal serves all 18; manual only +70285CA8060004,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 4: one SBC per metal serves all 18; manual only +70285CA8060005,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 5: one SBC per metal serves all 18; manual only +70285CA8060006,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 6: one SBC per metal serves all 18; manual only +70285CA8060007,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 7: one SBC per metal serves all 18; manual only +70285CA8060008,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 8: one SBC per metal serves all 18; manual only +70285CA8060009,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 9: one SBC per metal serves all 18; manual only +70285CA8060010,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 10: one SBC per metal serves all 18; manual only +70285CA8060011,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 11: one SBC per metal serves all 18; manual only +70285CA8060012,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 12: one SBC per metal serves all 18; manual only +70285CA8060014,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 14: one SBC per metal serves all 18; manual only +70285CA8060015,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 15: one SBC per metal serves all 18; manual only +70285CA8060016,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 16: one SBC per metal serves all 18; manual only +70285CA8060017,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 17: one SBC per metal serves all 18; manual only +70285CA8060018,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 18: one SBC per metal serves all 18; manual only +70285CA8060019,2026,https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName=Blue_Shield_Silver_70_Trio_HMO__MI011320_01-26_SBC.pdf,2026-09-26,Trio HMO rating area 19: one SBC per metal serves all 18; manual only +93689CA0110001,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IP_2IP_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IP&language=ENGL&productId=IEP&prescriptionPlanProductId=IEP,2026-09-26,rating area 2; Covered California product IEP; manual only +93689CA0150001,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IP_2IP_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IP&language=ENGL&productId=IEP&prescriptionPlanProductId=IEP,2026-09-26,rating area 3; Covered California product IEP; manual only +93689CA0110002,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IG_2IG_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IG&language=ENGL&productId=IEG&prescriptionPlanProductId=IEG,2026-09-26,rating area 2; Covered California product IEG; manual only +93689CA0150002,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IG_2IG_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IG&language=ENGL&productId=IEG&prescriptionPlanProductId=IEG,2026-09-26,rating area 3; Covered California product IEG; manual only +93689CA0120001,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IS_2IS_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IS&language=ENGL&productId=IES&prescriptionPlanProductId=IES,2026-09-26,rating area 2; Covered California product IES; manual only +93689CA0160001,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IS_2IS_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IS&language=ENGL&productId=IES&prescriptionPlanProductId=IES,2026-09-26,rating area 3; Covered California product IES; manual only +93689CA0120004,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IB_2IB_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IB&language=ENGL&productId=IEB&prescriptionPlanProductId=IEB,2026-09-26,rating area 2; Covered California product IEB; manual only +93689CA0160002,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2IB_2IB_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2IB&language=ENGL&productId=IEB&prescriptionPlanProductId=IEB,2026-09-26,rating area 3; Covered California product IEB; manual only +93689CA0120005,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2MC_2MC_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2MC&language=ENGL&productId=IEMC&prescriptionPlanProductId=IEMC,2026-09-26,rating area 2; Covered California product IEMC; manual only +93689CA0160003,2026,https://www.westernhealth.com/shop/?sp=plan-sbc&componentName=01-Jan-2026_2MC_2MC_EF_ENGL&sbcEffectiveDate=01/01/2026&medicalPlanID=2MC&language=ENGL&productId=IEMC&prescriptionPlanProductId=IEMC,2026-09-26,rating area 3; Covered California product IEMC; manual only +27603CA1150036,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemSilver70HMO/8XWT/English/DG166709956521.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XWT); unverified until downloaded; manual only +27603CA1150037,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemGold80HMO/8XXA/English/DG166709956514.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XXA); unverified until downloaded; manual only +27603CA1150038,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemPlatinum90HMO/8XWY/English/DG166709956741.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XWY); unverified until downloaded; manual only +27603CA1150039,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemBronze60HMO/8XW0/English/DG166709956513.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XW0); unverified until downloaded; manual only +27603CA1150040,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemMinimumCoverageHMO/8XWA/English/DG166709956740.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XWA); unverified until downloaded; manual only +27603CA1500005,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemGold80EPO/8XW6/English/DG166710600206.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XW6); unverified until downloaded; manual only +27603CA1500006,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemSilver70EPO/8XX9/English/DG166710600056.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XX9); unverified until downloaded; manual only +27603CA1500007,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemMinimumCoverageEPO/8XX0/English/DG166710600209.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XX0); unverified until downloaded; manual only +27603CA1500008,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemPlatinum90EPO/8XX1/English/DG166710600044.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XX1); unverified until downloaded; manual only +27603CA1500009,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemBronze60EPO/8XVZ/English/DG166710600179.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XVZ); unverified until downloaded; manual only +27603CA1500010,2026,https://sbc.anthem.com/dpsdeeplink/deepLink/AnthemBronze60HDHPEPO/8XXF/English/DG166710600207.pdf,2026-09-26,deep link inferred from the SBC posting site (contract 8XXF); unverified until downloaded; manual only +92815CA0010004,2026,https://www.lacare.org/sites/default/files/la4307_lacc_sbc_bronze_60_hmo_en_202508.pdf,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" +92815CA0010005,2026,https://www.lacare.org/sites/default/files/la4307_lacc_sbc_minimum_coverage_en_202508.pdf,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" +92815CA0010006,2026,https://www.lacare.org/sites/default/files/la4307_lacc_sbc_platinum_90_hmo_en_202508.pdf,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" +92815CA0010007,2026,https://www.lacare.org/sites/default/files/la4307_lacc_sbc_gold_80_hmo_en_202508.pdf,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" +92815CA0010008,2026,https://www.lacare.org/sites/default/files/la4307_lacc_sbc_silver_70_hmo_en_202508.pdf,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" +84014CA0010002,2026,https://files.santaclaracounty.gov/exjcpb1461/2025-10/sbc-2026-gold80-en.pdf?VersionId=I4NbNB5m6_8dmo67YyIEBTyqhJnXM8wP,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" +84014CA0010004,2026,https://files.santaclaracounty.gov/exjcpb1461/2025-10/sbc-2026-bronze60-en.pdf?VersionId=CjAIgv5nJE5CHcrC5hRxdLO1Jz_AXoOU,2026-09-26,"from a search result, never fetched; unverified until downloaded; manual only" diff --git a/src/ingestion/sbc/report.py b/src/ingestion/sbc/report.py index e8e7c3c..952bb30 100644 --- a/src/ingestion/sbc/report.py +++ b/src/ingestion/sbc/report.py @@ -18,6 +18,7 @@ from sqlalchemy import func, select from src.core.db import get_session +from src.core.exchanges import CATALOG_STATES from src.models.plan import Issuer, Plan from src.models.sbc import SbcDocument from src.services.sbc_status import READ_STATUSES, plan_sbc_status, sbc_document_join @@ -69,6 +70,7 @@ class Report: outdated: int # documents stored by an older parser stale_plans: dict[str, int] # state -> plans the latest run didn't return files: dict[str, tuple[int, int]] # folder -> (files, bytes) + acquired: Counter = field(default_factory=Counter) # crawl / manual -> documents read (ADR 0026) missing_files: list[str] = field(default_factory=list) changed_files: list[str] = field(default_factory=list) @@ -104,6 +106,11 @@ def collect(session, year: int, states: list[str] | None = None, verify: bool = ), stale_plans=_stale_plans(session, year, states), files=_files(year), + acquired=Counter(dict(session.execute( + select(SbcDocument.acquisition, func.count()) + .where(SbcDocument.plan_year == year, SbcDocument.status.in_(READ_STATUSES)) + .group_by(SbcDocument.acquisition) + ).all())), **_verify(session, year) if verify else {}, ) @@ -189,7 +196,8 @@ def render(report: Report) -> str: out += ["", "Why documents could not be read:"] out += [f" {count:>6} {status}: {detail}" for (status, detail), count in report.reasons.most_common()] - out += ["", f"Documents stored by an older parser (current is {PARSER_VERSION}): {report.outdated}"] + out += ["", f"Documents read: {report.acquired['crawl']} crawled, {report.acquired['manual']} imported by hand", + f"Documents stored by an older parser (current is {PARSER_VERSION}): {report.outdated}"] if report.orphans: out.append("Documents no plan points at (the plans moved link, or left): " + ", ".join(f"{count} {status}" for status, count in report.orphans.most_common())) @@ -207,12 +215,13 @@ def render(report: Report) -> str: def parse_args(args): parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--year", type=int, default=datetime.now(UTC).year) - parser.add_argument("--states", help="Comma-separated state codes, or ALL; default every state with plans") + parser.add_argument("--states", help="Comma-separated state codes (CA too), or ALL for the HealthCare.gov " + "states; default every state with plans") parser.add_argument("--verify-files", action="store_true", help="Also hash every stored PDF and report any missing or changed") parsed = parser.parse_args(args) try: - parsed.states = resolve_states(parsed.states) if parsed.states else None + parsed.states = resolve_states(parsed.states, CATALOG_STATES) if parsed.states else None except ValueError as exc: parser.error(str(exc)) return parsed diff --git a/src/ingestion/sources/registry.py b/src/ingestion/sources/registry.py index 5d4ba98..a29a706 100644 --- a/src/ingestion/sources/registry.py +++ b/src/ingestion/sources/registry.py @@ -8,12 +8,17 @@ from datetime import date from functools import cache from pathlib import Path +from urllib.parse import urlsplit REGISTRY_PATH = Path(__file__).with_name("registry.toml") KINDS = {"corpus", "catalog", "sbc_host", "directory"} -PERMISSIONS = {"public_domain", "open_license", "written_permission", "pending_review", "denied"} -APPROVED = {"public_domain", "open_license", "written_permission"} +# mandated_disclosure: an issuer's Summary of Benefits and Coverage, which +# federal law requires it to publish. Not a license: only an sbc_host may use +# it, and its commercial use stays under review (ADR 0026). +PERMISSIONS = {"public_domain", "open_license", "written_permission", "mandated_disclosure", "pending_review", + "denied"} +APPROVED = {"public_domain", "open_license", "written_permission", "mandated_disclosure"} COMMERCIAL_USE = {"yes", "no", "review"} ROBOTS = {"allowed", "disallowed", "unavailable", "not_applicable"} ACCESS = {"api", "download", "crawl", "manual", "link"} @@ -71,8 +76,18 @@ def _entry(raw: dict) -> RegisteredSource: raise RegistryError(f"{label}: verified_on must be a date") if not isinstance(raw["enabled"], bool): raise RegistryError(f"{label}: enabled must be true or false") + if raw["permission_status"] == "mandated_disclosure" and (raw["kind"] != "sbc_host" + or raw["commercial_use"] != "review"): + raise RegistryError(f"{label}: mandated_disclosure is only for an sbc_host, with commercial_use 'review'") if raw["enabled"] and raw["permission_status"] not in APPROVED: raise RegistryError(f"{label}: enabled needs permission_status in {sorted(APPROVED)}") + # A path after the host, so a prefix can never also match a look-alike + # host (https://a.example.com.evil.test); links are matched by prefix. + scopes = raw["scope_urls"] + if not (isinstance(scopes, list) and scopes and all( + isinstance(prefix, str) and prefix.startswith("https://") and "/" in prefix[len("https://"):] + and urlsplit(prefix).netloc for prefix in scopes)): + raise RegistryError(f"{label}: scope_urls {scopes!r} must be a list of https://host/… prefixes") jurisdiction = raw["jurisdiction"] if not (isinstance(jurisdiction, str) and len(jurisdiction) == 2 and jurisdiction.isalpha() and jurisdiction.isupper()): @@ -113,3 +128,20 @@ def require_enabled(source_id: str) -> RegisteredSource: if not entry.enabled: raise SourceNotApprovedError(f"{source_id} is disabled ({entry.permission_status})") return entry + + +def sbc_host_for(url: str) -> RegisteredSource | None: + """The sbc_host entry whose scope covers this URL, enabled or not; None if none does. + + Scopes are URL prefixes; the longest match wins, so a narrower entry can + sit inside a broader one. + """ + matches = [(len(prefix), entry) for entry in _committed().values() if entry.kind == "sbc_host" + for prefix in entry.scope_urls if url.startswith(prefix)] + return max(matches, key=lambda match: match[0])[1] if matches else None + + +def host_access(url: str) -> str | None: + """How an SBC link may be read: its enabled sbc_host's access ("crawl", "manual"), or None (ADR 0026).""" + entry = sbc_host_for(url) + return entry.access if entry is not None and entry.enabled else None diff --git a/src/ingestion/sources/registry.toml b/src/ingestion/sources/registry.toml index a51c636..92ed8f2 100644 --- a/src/ingestion/sources/registry.toml +++ b/src/ingestion/sources/registry.toml @@ -1,6 +1,8 @@ # Every source PolicyPal reads or links, and whether it may be used (ADR 0025). -# `enabled = true` needs permission_status public_domain, open_license or -# written_permission; registry.py refuses anything else when it loads. +# `enabled = true` needs permission_status public_domain, open_license, +# written_permission or mandated_disclosure; registry.py refuses anything else +# when it loads. mandated_disclosure is only for an sbc_host (an issuer's SBC, +# which federal law requires it to publish), with commercial_use "review" (ADR 0026). # Record a refused source too, disabled, so the decision is not re-litigated. [[source]] @@ -113,3 +115,220 @@ verified_on = 2026-09-24 enabled = false removal = "Nothing is stored; remove the links in src/core/exchanges.py and frontend/src/features/plans/exchanges.js." notes = "Linked, never read. Its Terms of Use forbid access 'through any automated means (including, but not limited to, use of scripts, web crawlers, or screen scrapers)' and downloading or republishing without a separate written agreement. robots.txt allows everything; the Terms govern." + +# California carriers' SBCs (ADR 0026). Each is a federally mandated disclosure, +# not a licensed work, so commercial use stays under review. A carrier whose +# website Terms ban robots or scrapers is manual-only, whatever its robots.txt +# says: a person downloads its SBCs in a browser and `make import-sbc` reads +# them. Removal for every one: set enabled = false, then +# `make apply-sbc-manifest YEAR=…`; its plans lose their links, and its PDFs +# are kept (ADR 0016). Evidence per carrier: docs/findings/ca-sbc.md. + +[[source]] +id = "sbc_iehp" +name = "IEHP Covered California SBCs" +publisher = "Inland Empire Health Plan (a public entity)" +scope_urls = ["https://www.iehp.org/content/dam/iehp-org/en/documents/coveredcalifornia/summary-of-benefits-and-coverage/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved (site footer); no website terms of use" +license_url = "https://www.iehp.org/en/terms-and-conditions" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "crawl" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Static PDFs named by metal level, listed at https://www.iehp.org/en/browse-plans/covered-california. The linked terms page covers only IEHP's text-message service; no website terms were found, so nothing bans automated access." + +[[source]] +id = "sbc_molina_ca" +name = "Molina Healthcare of California Marketplace SBCs" +publisher = "Molina Healthcare of California" +scope_urls = ["https://www.molinamarketplace.com/marketplace/ca/en-us/-/media/Molina/PublicWebsite/PDF/members/ca/en-us/Marketplace/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; use limited to learning about Molina's plans" +license_url = "https://www.molinahealthcare.com/members/common/en-us/terms_privacy.aspx" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "crawl" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Files named CASBCE__1.pdf, beside the EOCs listed at https://www.molinamarketplace.com/marketplace/ca/en-us/MemberForms.aspx (the page lists only EOCs). The Terms have no robots clause, but say 'You may use the Sites solely for the purposes of learning about ... Molina Healthcare's health plans' and bar reproduction or distribution 'for any other purpose'." + +[[source]] +id = "sbc_sharp" +name = "Sharp Health Plan Covered California SBCs" +publisher = "Sharp Health Plan" +scope_urls = ["https://www.sharphealthplan.com/docs/default-source/plan-documents/summary-of-benefits-and-coverage/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; no reproduction or redistribution beyond personal use without written consent" +license_url = "https://www.sharphealthplan.com/terms-of-use-and-registration-agreement" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "crawl" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Listed at https://www.sharphealthplan.com/our-plans/individual-and-family-plans/plan-details-rates-2026. robots.txt disallows /docs/default-source/pdfs/, which this folder is not under. The Terms have no robots clause, only 'not to misuse, abuse, or overuse beyond reasonable amounts'. Each SBC prints its HIOS plan ID." + +[[source]] +id = "sbc_balance_cchp" +name = "Balance by CCHP Covered California SBCs" +publisher = "Chinese Community Health Plan (Balance by CCHP)" +scope_urls = ["https://balancebycchp.com/wp-content/uploads/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; stored only for personal, non-commercial use; no republication without written consent" +license_url = "https://balancebycchp.com/terms-and-conditions/" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "crawl" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "The Terms have no robots clause. On 2026-09-26 the site held 2026 EOCs but only 2025 SBCs, so no 2026 plan is linked; re-check when writing each year's manifest." + +[[source]] +id = "sbc_kaiser_ca" +name = "Kaiser Permanente California individual and family SBCs" +publisher = "Kaiser Foundation Health Plan, Inc." +scope_urls = ["https://healthy.kaiserpermanente.org/content/dam/kporg/final/documents/health-plan-documents/summary-of-benefits/ca/individual-family/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; a single copy for personal, noncommercial use" +license_url = "https://healthy.kaiserpermanente.org/termsconditions" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "manual" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Manual only: under the Terms, using any 'robot', 'spider', data mining tools ... or other automatic device ... to (1) access, acquire, copy or monitor any portion of the Site is a material breach. SBCs are named -01-en-.pdf." + +[[source]] +id = "sbc_blue_shield_ca" +name = "Blue Shield of California individual and family SBCs" +publisher = "California Physician's Service DBA Blue Shield of California" +scope_urls = ["https://www.blueshieldca.com/memberwebapp/unauth-document-download?fileName="] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; personal, non-commercial use only" +license_url = "https://www.blueshieldca.com/en/home/site-help/terms-of-use" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "manual" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Manual only: the Terms forbid accessing the site 'with any engine, software, tool, agent, device or mechanism other than' Blue Shield's own or 'generally available third party web browsers'. Listed at https://www.blueshieldca.com/memberwebapp/welcome/documents/ifp; there is no static PDF URL. One Trio HMO SBC per metal serves all 18 regional plan IDs. robots.txt disallows */download/." + +[[source]] +id = "sbc_western_health_advantage" +name = "Western Health Advantage Covered California SBCs" +publisher = "Western Health Advantage" +scope_urls = ["https://www.westernhealth.com/shop/?sp=plan-sbc"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; a personal licence to access the site" +license_url = "https://www.westernhealth.com/legal/terms-of-use/" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "manual" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Manual only: the Terms say 'You may not ... (2) use any data mining, robots, or similar gathering and extraction tools'. Their reproduction ban exempts 'files containing Benefit Plans and Sales Materials', which may cover SBCs; not relied on. SBCs are generated from a query URL; one serves each plan's pair of rating-area filings. robots.txt asks for Crawl-Delay: 5." + +[[source]] +id = "sbc_anthem_ca" +name = "Anthem Blue Cross California SBCs" +publisher = "Anthem Blue Cross" +scope_urls = ["https://sbc.anthem.com/dpsdeeplink/deepLink/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; for your own use, no commercial purpose" +license_url = "https://www.anthem.com/ca/terms-of-use" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "unavailable" +robots_checked_on = 2026-09-26 +access = "manual" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Manual only: the Terms forbid any 'robot, spider, site search/retrieval application or other manual or automatic device to retrieve, index, \"scrape,\" \"data mine\"', and creating 'a database by systematically downloading'. sbc.anthem.com/robots.txt answers 401; www.anthem.com's disallows /*.pdf$. Plans are found on the SBC posting site, https://sbc.anthem.com/dps. The 30-state crawl also reads this host (ADR 0026: a follow-up)." + +[[source]] +id = "sbc_la_care" +name = "L.A. Care Covered California SBCs" +publisher = "L.A. Care Health Plan" +scope_urls = ["https://www.lacare.org/sites/default/files/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; commercial use prohibited without written authorization (from search results; the page was not fetched)" +license_url = "https://www.lacare.org/terms-and-conditions" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-25 +access = "manual" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Manual only, and never fetched by any script: robots.txt names AI crawlers (anthropic-ai, ClaudeBot, ChatGPT-User, …) as unwelcome, taken as intent, though `*` allows us. Search results show the Terms banning robots and scrapers. Listed at https://lacare.org/members/documents/la-care-covered; la5600_laccd_* files are the off-exchange line." + +[[source]] +id = "sbc_valley_health_plan" +name = "Valley Health Plan Covered California SBCs" +publisher = "Valley Health Plan (County of Santa Clara)" +scope_urls = ["https://files.santaclaracounty.gov/exjcpb1461/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "Unknown: the site blocks automated clients, so its terms could not be read" +license_url = "https://www.valleyhealthplan.org/" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "unavailable" +robots_checked_on = 2026-09-26 +access = "manual" +verified_on = 2026-09-26 +enabled = true +removal = "Set enabled = false, then `make apply-sbc-manifest YEAR=…`." +notes = "Manual only: valleyhealthplan.org and the county's file host both answer automated clients with a Cloudflare 403. Listed at https://www.valleyhealthplan.org/benefits-and-coverage-information-covered-california-and-ifp-members." + +[[source]] +id = "sbc_health_net_ca" +name = "Health Net California SBCs" +publisher = "Health Net of California" +scope_urls = ["https://www.healthnet.com/portal/shopping/"] +kind = "sbc_host" +jurisdiction = "CA" +license = "All rights reserved; downloads for personal, non-commercial use only" +license_url = "https://www.healthnet.com/content/healthnet/en_us/disclaimers/legal/terms-conditions.html" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-25 +access = "manual" +verified_on = 2026-09-26 +enabled = false +removal = "Already disabled: its plans have no link." +notes = "Disabled (user decision, 2026-09-25): SBCs are reachable only through a POST search form (/portal/shopping/searchSBC.action), so its 61 plans show no link and point to Covered California." diff --git a/src/models/sbc.py b/src/models/sbc.py index e79fea8..dcac492 100644 --- a/src/models/sbc.py +++ b/src/models/sbc.py @@ -21,6 +21,9 @@ # A plan's view of its document (ADR 0017): no link to one, a link never # read, or the document's own status. PLAN_SBC_STATUSES = ("no_link", "not_read", *SBC_STATUSES) +# How the PDF reached us: fetched by the crawler, or downloaded by a person and +# imported with `make import-sbc` (ADR 0026). A manual one is never requested. +SBC_ACQUISITIONS = ("crawl", "manual") class SbcDocument(Base): @@ -39,6 +42,7 @@ class SbcDocument(Base): __table_args__ = ( UniqueConstraint("url", "plan_year", name="uq_sbc_documents_url_plan_year"), CheckConstraint(f"status IN {SBC_STATUSES}", name="ck_sbc_documents_status"), + CheckConstraint(f"acquisition IN {SBC_ACQUISITIONS}", name="ck_sbc_documents_acquisition"), ) id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) @@ -60,6 +64,7 @@ class SbcDocument(Base): etag: Mapped[str | None] = mapped_column(String(200), nullable=True) last_modified: Mapped[str | None] = mapped_column(String(64), nullable=True) checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True) + acquisition: Mapped[str] = mapped_column(String(8), nullable=False, server_default="crawl") created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), nullable=False) updated_at: Mapped[datetime] = mapped_column( DateTime(timezone=True), server_default=func.now(), onupdate=func.now(), nullable=False diff --git a/src/services/generation.py b/src/services/generation.py index 7f28b6f..c573319 100644 --- a/src/services/generation.py +++ b/src/services/generation.py @@ -469,7 +469,7 @@ def answer(query: str, chunks: list[RetrievedChunk], if coverage_read: cited = cited_labels(answer_text) chunks = [c for c in chunks if c.source in cited] - # Server-written, so it is said exactly when it is true (ADR 0024). + # Server-written, so it is said exactly when it is true (ADR 0024, 0026). answer_text = "\n\n".join([*notices, answer_text]) return Answer(answer_text, _distinct(chunks + passages), tuple(plans.values()), needs) diff --git a/src/services/plan_coverage.py b/src/services/plan_coverage.py index 87dccc9..ae82c37 100644 --- a/src/services/plan_coverage.py +++ b/src/services/plan_coverage.py @@ -41,6 +41,7 @@ class PlanCoverage: passages: tuple[RetrievedChunk, ...] = () # The precise status behind `status` (ADR 0017): "blocked", "not_read"… sbc_status: str | None = None + state: str | None = None def coverage_for(session, plan_ids: list[str], question: str, @@ -56,7 +57,7 @@ def coverage_for(session, plan_ids: list[str], question: str, def _coverage(session, plan_id: str, question: str, year: int | None) -> PlanCoverage: stmt = ( select(Plan.marketing_name, Plan.plan_year, Plan.benefits_url, Issuer.name, - SbcDocument.id, plan_sbc_status()) + SbcDocument.id, plan_sbc_status(), Plan.state) .join(Issuer, Issuer.id == Plan.issuer_id) .outerjoin(SbcDocument, sbc_document_join()) .where(Plan.hios_plan_id == plan_id) @@ -69,10 +70,10 @@ def _coverage(session, plan_id: str, question: str, year: int | None) -> PlanCov if row is None: return PlanCoverage(plan_id, "not_found") - name, plan_year, url, issuer, document_id, sbc_status = row + name, plan_year, url, issuer, document_id, sbc_status, state = row if sbc_status not in READ_STATUSES: status = "no_document" if sbc_status in ("no_link", "not_read") else "unavailable" - return PlanCoverage(plan_id, status, name, issuer, plan_year, url, sbc_status=sbc_status) + return PlanCoverage(plan_id, status, name, issuer, plan_year, url, sbc_status=sbc_status, state=state) chunks = session.execute( select(SbcChunk.chunk_id, SbcChunk.section, SbcChunk.content) @@ -91,7 +92,7 @@ def _coverage(session, plan_id: str, question: str, year: int | None) -> PlanCov ) # Scores are logged, not gated on (ADR 0014); never the question itself. logger.info("plan_coverage %s: %s", plan_id, [round(p.score, 3) for p in passages]) - return PlanCoverage(plan_id, "ok", name, issuer, plan_year, url, passages, sbc_status) + return PlanCoverage(plan_id, "ok", name, issuer, plan_year, url, passages, sbc_status, state) def source_label(plan_name: str, section: str) -> str: diff --git a/src/services/tools.py b/src/services/tools.py index 4710efe..b0b5a3d 100644 --- a/src/services/tools.py +++ b/src/services/tools.py @@ -13,7 +13,7 @@ from pydantic import BaseModel, ConfigDict, Field, StringConstraints, ValidationError from src.core.db import get_session -from src.core.exchanges import exchange_for +from src.core.exchanges import FILED_RATE_STATES, exchange_for from src.core.logging import get_logger from src.core.marketplace_api import REFERENCE_AGE @@ -248,6 +248,22 @@ def _render(result: PlanSearchResult) -> ToolOutcome: return ToolOutcome(json.dumps(payload, default=str), plans=result.plans, notice=_prior_year_notice(result)) +def _standard_version_notice(coverages: list[PlanCoverage]) -> str | None: + """That a filed-rate state's SBCs are its plans' standard versions, when one is quoted (ADR 0026). + + Those links are chosen by hand, one document per base plan, so what the + cost-sharing variants charge is not in it. Written here, like the + prior-year notice: asked to repeat a note, the model left it out. + """ + exchanges = sorted({FILED_RATE_STATES[c.state].name for c in coverages + if c.passages and c.state in FILED_RATE_STATES}) + if not exchanges: + return None + return (f"Summaries of Benefits and Coverage quoted here for {' and '.join(exchanges)} plans are for each " + "plan's standard version; people who qualify for cost-sharing reductions or American Indian and " + "Alaska Native cost sharing pay less than they show.") + + def _coverage_row(coverage: PlanCoverage) -> dict: row = {"plan_id": coverage.plan_id, "status": coverage.status} if coverage.status == "not_found": @@ -342,4 +358,4 @@ def _plan_coverage(raw_arguments: str, plan_years: Mapping[str, int] | None) -> logger.info("plan_coverage: %s", [c.status for c in coverages]) payload = {"status": "ok", "plans": [_coverage_row(c) for c in coverages]} chunks = tuple(p for c in coverages for p in c.passages) - return ToolOutcome(json.dumps(payload, default=str), chunks=chunks) + return ToolOutcome(json.dumps(payload, default=str), chunks=chunks, notice=_standard_version_notice(coverages)) diff --git a/tests/conftest.py b/tests/conftest.py index 60aba5e..41169c7 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -59,3 +59,48 @@ def session(): db.close() transaction.rollback() connection.close() + + +_SBC_HOST = """ +[[source]] +id = "{id}" +name = "{id}" +publisher = "An insurer" +scope_urls = ["{prefix}"] +kind = "sbc_host" +jurisdiction = "{state}" +license = "All rights reserved" +license_url = "https://example.com/terms" +permission_status = "mandated_disclosure" +commercial_use = "review" +robots = "allowed" +robots_checked_on = 2026-09-26 +access = "{access}" +verified_on = 2026-09-26 +enabled = {enabled} +removal = "Disable it." +notes = "" +""" + + +@pytest.fixture +def sbc_hosts(monkeypatch, tmp_path): + """The committed registry plus made-up carrier hosts (ADR 0026). + + Returns {id: URL prefix}: `crawl` and `manual` are enabled California + hosts, `off` a disabled one, and `texas` an enabled host of another state. + """ + from src.ingestion.sources import registry + + hosts = { + "crawl": ("https://crawl.example.com/sbc/", "CA", "crawl", "true"), + "manual": ("https://manual.example.com/sbc/", "CA", "manual", "true"), + "off": ("https://off.example.com/sbc/", "CA", "crawl", "false"), + "texas": ("https://texas.example.com/sbc/", "TX", "crawl", "true"), + } + path = tmp_path / "registry.toml" + path.write_text("".join(_SBC_HOST.format(id=id_, prefix=prefix, state=state, access=access, enabled=enabled) + for id_, (prefix, state, access, enabled) in hosts.items()), encoding="utf-8") + loaded = registry.load_registry() | registry.load_registry(path) + monkeypatch.setattr(registry, "_committed", lambda: loaded) + return {id_: prefix for id_, (prefix, *_) in hosts.items()} diff --git a/tests/test_ca_puf_load.py b/tests/test_ca_puf_load.py index eb23484..06decbc 100644 --- a/tests/test_ca_puf_load.py +++ b/tests/test_ca_puf_load.py @@ -311,3 +311,52 @@ def test_an_empty_file_changes_nothing(tmp_path, session): _load(session, empty) assert _counts(session) == before + + +def _manifest(tmp_path, monkeypatch, session, *lines): + """The year's SBC manifest, applied in the test's transaction (ADR 0026).""" + from src.ingestion.sbc import manifest + + folder = tmp_path / "manifests" + folder.mkdir() + (folder / f"ca-{YEAR}.csv").write_text("\n".join(["hios_plan_id,plan_year,sbc_url,verified_on,note", *lines]), + encoding="utf-8") + monkeypatch.setattr(manifest, "MANIFEST_DIR", folder) + monkeypatch.setattr(manifest, "get_session", _in(session)) + + +def test_the_command_applies_the_years_sbc_manifest_after_loading(tmp_path, session, capsys, monkeypatch, sbc_hosts): + _crosswalk(session) + _manifest(tmp_path, monkeypatch, session, f"{KAISER},{YEAR},{sbc_hosts['crawl']}kaiser.pdf,2026-09-26,") + + with patch.object(cli, "get_session", _in(session)): + cli.main(["--year", str(YEAR), "--zip", str(_zip(tmp_path))]) + + assert f"1 California plans for {YEAR} linked to 1 SBC documents" in capsys.readouterr().out + assert session.scalar(select(Plan.benefits_url).where(Plan.hios_plan_id == KAISER)) == ( + f"{sbc_hosts['crawl']}kaiser.pdf") + + +def test_a_broken_manifest_leaves_the_plans_loaded_and_says_the_links_were_not_applied( + tmp_path, session, monkeypatch, sbc_hosts): + _crosswalk(session) + _manifest(tmp_path, monkeypatch, session, f"{KAISER},{YEAR},https://nobody.example.com/k.pdf,2026-09-26,") + + with patch.object(cli, "get_session", _in(session)), pytest.raises(SystemExit) as exit_: + cli.main(["--year", str(YEAR), "--zip", str(_zip(tmp_path))]) + + assert "Plans loaded, but their SBC links were not applied: line 2" in str(exit_.value.code) + assert session.scalar(select(Plan.benefits_url).where(Plan.hios_plan_id == KAISER, Plan.plan_year == YEAR)) is None + + +def test_without_a_manifest_the_command_says_every_plan_has_no_link(tmp_path, session, capsys, monkeypatch): + from src.ingestion.sbc import manifest + + _crosswalk(session) + monkeypatch.setattr(manifest, "MANIFEST_DIR", tmp_path) + + with patch.object(cli, "get_session", _in(session)): + cli.main(["--year", str(YEAR), "--zip", str(_zip(tmp_path))]) + + assert f"No SBC manifest for {YEAR} (ca-{YEAR}.csv): every plan is shown with no SBC link" in ( + capsys.readouterr().out) diff --git a/tests/test_ingestion_plans.py b/tests/test_ingestion_plans.py index 0b5b8a7..47d41c9 100644 --- a/tests/test_ingestion_plans.py +++ b/tests/test_ingestion_plans.py @@ -209,3 +209,13 @@ def test_asking_the_api_loader_for_california_names_its_own_loader(): with pytest.raises(ValueError) as other: resolve_states("NY") assert "ingest-ca-plans" not in str(other.value) + + +def test_a_command_that_reads_california_too_can_allow_it(): + """The SBC ingest and report pass CATALOG_STATES; ALL stays the 30 API states (ADR 0026).""" + from src.core.exchanges import CATALOG_STATES + + assert resolve_states("ca, tx", CATALOG_STATES) == ["CA", "TX"] + assert "CA" not in resolve_states("ALL", CATALOG_STATES) + with pytest.raises(ValueError, match="not states PolicyPal has plans for here: NY"): + resolve_states("NY", CATALOG_STATES) diff --git a/tests/test_plan_coverage.py b/tests/test_plan_coverage.py index 8937992..244f706 100644 --- a/tests/test_plan_coverage.py +++ b/tests/test_plan_coverage.py @@ -144,3 +144,12 @@ def test_a_long_plan_name_gives_way_so_the_label_fits(): assert len(label) == 300 assert label.endswith(" - Summary of Benefits - If you have a test.pdf") + + +def test_a_plans_state_comes_back_with_its_coverage(session, issuer): + """Whether the standard-version notice applies is decided from it (ADR 0026).""" + _plan(session, issuer, "99999NH0010001", SHARED) + _document(session, SHARED, test="Imaging $100 copay") + _plan(session, issuer, "99999NH0010002", None) + + assert [c.state for c in coverage_for(session, ["99999NH0010001", "99999NH0010002"], "MRI?")] == ["NH", "NH"] diff --git a/tests/test_registry.py b/tests/test_registry.py index 37c8697..fbd61c5 100644 --- a/tests/test_registry.py +++ b/tests/test_registry.py @@ -9,6 +9,7 @@ SourceNotApprovedError, load_registry, require_enabled, + sbc_host_for, ) _ENTRY = """ @@ -130,3 +131,63 @@ class _Unregistered(_Denied): name = registry_id = "nobody_approved_this" assert enabled_sources([_Denied(), _Unregistered()]) == [] + + +def _sbc_host(id_="h", prefix="https://sbc.example.com/", commercial="review", enabled="true", kind="sbc_host"): + return (_entry(id_, "mandated_disclosure", enabled) + .replace('kind = "corpus"', f'kind = "{kind}"') + .replace('commercial_use = "yes"', f'commercial_use = "{commercial}"') + .replace('scope_urls = ["https://example.com/"]', f'scope_urls = ["{prefix}"]')) + + +def test_a_mandated_disclosure_is_approved_only_for_an_sbc_host_under_review(tmp_path): + """An SBC is published because the law requires it; that is not a licence (ADR 0026).""" + assert load_registry(_write(tmp_path, _sbc_host()))["h"].enabled + + for broken in (_sbc_host(kind="corpus"), _sbc_host(commercial="yes"), _sbc_host(commercial="no")): + with pytest.raises(RegistryError, match="h: mandated_disclosure is only for an sbc_host"): + load_registry(_write(tmp_path, broken)) + + +def test_the_sbc_host_is_the_longest_prefix_covering_the_link_enabled_or_not(tmp_path, monkeypatch): + from src.ingestion.sources import registry + + loaded = load_registry(_write(tmp_path, "".join([ + _sbc_host("broad", "https://sbc.example.com/"), + _sbc_host("narrow", "https://sbc.example.com/ca/", enabled="false"), + _entry("corpus").replace('scope_urls = ["https://example.com/"]', 'scope_urls = ["https://corpus.example.com/"]'), + ]))) + monkeypatch.setattr(registry, "_committed", lambda: loaded) + + assert sbc_host_for("https://sbc.example.com/ca/gold.pdf").id == "narrow" + assert sbc_host_for("https://sbc.example.com/tx/gold.pdf").id == "broad" + assert sbc_host_for("https://corpus.example.com/gold.pdf") is None # not an sbc_host + assert sbc_host_for("https://sbc.example.com.evil.test/gold.pdf") is None + + +def test_every_california_carrier_is_a_mandated_disclosure_under_review(): + hosts = {id_: e for id_, e in load_registry().items() if e.kind == "sbc_host" and e.jurisdiction == "CA"} + + assert len(hosts) == 11 + for entry in hosts.values(): + assert (entry.permission_status, entry.commercial_use) == ("mandated_disclosure", "review") + assert all(prefix.startswith("https://") for prefix in entry.scope_urls) + + +def test_carriers_whose_terms_ban_robots_are_never_crawled(): + """The user's rule (ADR 0026): website Terms govern, as for Covered California.""" + registry = load_registry() + + for id_ in ("sbc_kaiser_ca", "sbc_blue_shield_ca", "sbc_western_health_advantage", "sbc_anthem_ca", + "sbc_la_care", "sbc_valley_health_plan"): + assert registry[id_].access == "manual", id_ + assert not registry["sbc_health_net_ca"].enabled + + +@pytest.mark.parametrize("scope", ['"https://sbc.example.com/"', '["https://sbc.example.com"]', + '["http://sbc.example.com/"]', '["sbc.example.com/"]', '[42]']) +def test_a_scope_must_be_https_with_a_path_after_the_host(tmp_path, scope): + """Links are matched by prefix: without the "/", https://sbc.example.com would also match + https://sbc.example.com.evil.test/…""" + with pytest.raises(RegistryError, match="x: scope_urls .* must be a list of https://host/"): + load_registry(_write(tmp_path, _entry().replace('["https://example.com/"]', scope))) diff --git a/tests/test_sbc_fetch.py b/tests/test_sbc_fetch.py index ec21707..b6de078 100644 --- a/tests/test_sbc_fetch.py +++ b/tests/test_sbc_fetch.py @@ -13,7 +13,9 @@ NOT_MODIFIED, cache_path, fetch_pdf, + looks_like_pdf, revalidate, + unsafe_reason, ) URL = "https://sbc.example.com/plans/2026/gold.pdf" @@ -224,3 +226,19 @@ def test_a_network_error_is_worth_trying_again(_offline): assert (result.status, result.transient) == ("http_error", True) assert "12345" not in (result.detail or "") + + +@pytest.mark.parametrize(("head", "pdf"), [ + (b"%PDF-1.7\n", True), + (b"\xef\xbb\xbf \n%PDF-1.4", True), # a little junk before the header, as some servers send + (b"Sign in", False), + (b" " * 2048 + b"%PDF-1.7", False), # past the window +]) +def test_a_pdf_is_known_by_its_header_near_the_start(head, pdf): + """Shared by the fetcher and `make import-sbc` (ADR 0026).""" + assert looks_like_pdf(head) is pdf + + +def test_the_safety_check_is_public_for_manifest_links(): + assert unsafe_reason("https://sbc.example.com/gold.pdf") is None + assert unsafe_reason("http://sbc.example.com/gold.pdf") is not None diff --git a/tests/test_sbc_import.py b/tests/test_sbc_import.py new file mode 100644 index 0000000..cd7fd48 --- /dev/null +++ b/tests/test_sbc_import.py @@ -0,0 +1,257 @@ +"""`make import-sbc`: reading an SBC a person downloaded in a browser (ADR 0026). + +Real Postgres through the rolled-back `session` fixture; the real fetch cache, +pointed at a temporary folder, with the network replaced by a mock that +fails the test if anything asks it. The parser is the real one, fed +synthetic pages. +""" +from contextlib import contextmanager +from pathlib import Path +from unittest.mock import MagicMock + +import pytest +from sqlalchemy import select + +from src.ingestion.ca_puf.load import CATALOG_SOURCE +from src.ingestion.sbc import fetch, import_pdf, ingest +from src.ingestion.sbc.extract import ( + EVENT_HEADINGS, + QUESTION_HEADINGS, + PdfPage, + TableRow, + parse_sbc, +) +from src.ingestion.sbc.import_pdf import SbcImportError, import_file, import_folder +from src.models.plan import Issuer, Plan +from src.models.sbc import SbcChunk, SbcDocument + +YEAR = 1999 +GOLD_PDF = b"%PDF-1.7 gold" + + +def _pages(test_cost="$60", year=YEAR): + header = f"Coverage Period: 01/01/{year}-12/31/{year}\n: Example Gold | Coverage for: Individual" + rows = [("Important Questions", "Answers | Why This Matters:")] + [ + (heading, "No.") for heading in QUESTION_HEADINGS] + [("Common Medical Event", "")] + [ + (heading, f"Imaging {test_cost} copay" if heading == "If you have a test" else "No charge") + for heading in EVENT_HEADINGS] + return [PdfPage(text=header, rows=tuple(TableRow(label, body) for label, body in rows))] + + +@pytest.fixture +def network(monkeypatch, tmp_path): + """Any request at all fails the test: an import reads only what it is given.""" + get = MagicMock(side_effect=AssertionError("an import made a request")) + monkeypatch.setattr(fetch.requests, "get", get) + return get + + +@pytest.fixture +def pages(monkeypatch, session, tmp_path, network): + """The parser's input for each file's bytes; returns the dict to add to.""" + served = {GOLD_PDF: _pages()} + + @contextmanager + def same_session(): + yield session + session.flush() + + monkeypatch.setattr(fetch, "SBC_RAW", tmp_path / "raw") + monkeypatch.setattr(ingest, "SBC_ARCHIVE", tmp_path / "archive") + monkeypatch.setattr(ingest, "SBC_REJECTED", tmp_path / "rejected") + monkeypatch.setattr(ingest, "get_session", same_session) + monkeypatch.setattr(import_pdf, "get_session", same_session) + monkeypatch.setattr(ingest, "read_parsed", lambda path: (served[path.read_bytes()], + parse_sbc(served[path.read_bytes()]))) + return served + + +@pytest.fixture +def links(session, sbc_hosts): + """California plans linked to one manual-only, one crawlable and one disabled host.""" + issuer = Issuer(hios_issuer_id="11111", plan_year=YEAR, name="Example Care", state="CA") + session.add(issuer) + session.flush() + urls = { + "manual": f"{sbc_hosts['manual']}gold.pdf", + "query": f"{sbc_hosts['manual']}download?fileName=Silver_70_SBC.pdf", + "crawl": f"{sbc_hosts['crawl']}bronze.pdf", + "off": f"{sbc_hosts['off']}platinum.pdf", + } + for number, url in enumerate(urls.values(), 1): + session.add(Plan(issuer_id=issuer.id, hios_plan_id=f"11111CA001000{number}", plan_year=YEAR, + marketing_name="Gold 80 HMO", metal_level="Gold", plan_type="HMO", state="CA", + benefits_url=url, hsa_eligible=False, has_national_network=False, + catalog_source=CATALOG_SOURCE)) + session.flush() + return urls + + +def _download(folder: Path, name="gold.pdf", body=GOLD_PDF) -> Path: + folder.mkdir(parents=True, exist_ok=True) + (folder / name).write_bytes(body) + return folder / name + + +def _document(session, url): + return session.scalar(select(SbcDocument).where(SbcDocument.url == url, SbcDocument.plan_year == YEAR)) + + +def _pdfs(tmp_path): + return sorted(path.read_bytes() for path in (tmp_path / "raw").glob("**/*.pdf")) + sorted( + path.read_bytes() for path in (tmp_path / "archive").glob("**/*.pdf")) + + +def test_a_downloaded_sbc_is_read_as_manual_with_no_request(pages, links, session, tmp_path, network): + downloaded = _download(tmp_path / "downloads") + + assert import_file(downloaded, links["manual"], YEAR) == "ok" + + document = _document(session, links["manual"]) + assert (document.status, document.acquisition, document.title) == ("ok", "manual", "Example Gold") + assert session.scalars(select(SbcChunk.section).where(SbcChunk.document_id == document.id)).all() + assert fetch.cache_path(links["manual"], YEAR).read_bytes() == GOLD_PDF + assert downloaded.read_bytes() == GOLD_PDF # the person's own file is left where it was + network.assert_not_called() + + +def test_a_corrected_download_archives_the_file_it_replaces(pages, links, session, tmp_path): + import_file(_download(tmp_path / "downloads"), links["manual"], YEAR) + pages[b"%PDF-1.7 gold v2"] = _pages(test_cost="$75") + + assert import_file(_download(tmp_path / "downloads", body=b"%PDF-1.7 gold v2"), links["manual"], YEAR) == "ok" + + assert _pdfs(tmp_path) == [b"%PDF-1.7 gold v2", GOLD_PDF] + assert _document(session, links["manual"]).acquisition == "manual" + + +def test_importing_the_same_file_again_archives_nothing(pages, links, tmp_path): + for _ in range(2): + import_file(_download(tmp_path / "downloads"), links["manual"], YEAR) + + assert _pdfs(tmp_path) == [GOLD_PDF] + + +def test_a_download_for_another_year_is_refused_and_kept_aside(pages, links, session, tmp_path): + pages[b"%PDF-1.7 last year"] = _pages(year=YEAR - 1) + + assert import_file(_download(tmp_path / "downloads", body=b"%PDF-1.7 last year"), links["manual"], + YEAR) == "wrong_year" + + assert _document(session, links["manual"]).acquisition == "manual" + assert [p.read_bytes() for p in (tmp_path / "rejected").glob("**/*.pdf")] == [b"%PDF-1.7 last year"] + + +@pytest.mark.parametrize(("link", "body", "message"), [ + ("https://manual.example.com/sbc/nobody-links-this.pdf", GOLD_PDF, "no 1999 plan links to"), + ("off", GOLD_PDF, "has no enabled sbc_host entry"), + ("manual", b"Sign in", "is not a PDF"), +]) +def test_a_file_that_cannot_be_imported_writes_nothing(pages, links, session, tmp_path, link, body, message): + url = links.get(link, link) + + with pytest.raises(SbcImportError, match=message): + import_file(_download(tmp_path / "downloads", body=body), url, YEAR) + + assert _document(session, url) is None + assert not (tmp_path / "raw").exists() + + +def test_a_file_past_the_size_cap_is_refused(pages, links, session, tmp_path, monkeypatch): + monkeypatch.setattr(import_pdf, "MAX_BYTES", len(GOLD_PDF) - 1) + + with pytest.raises(SbcImportError, match="not a Summary of Benefits"): + import_file(_download(tmp_path / "downloads"), links["manual"], YEAR) + + +def test_a_path_that_is_not_a_file_is_refused(pages, links, tmp_path): + with pytest.raises(SbcImportError, match="is not a file"): + import_file(tmp_path / "missing.pdf", links["manual"], YEAR) + + +def test_a_folder_is_matched_to_manual_links_by_file_name(pages, links, session, tmp_path): + """By the link's last path segment, or a query value naming the PDF (Blue Shield's ?fileName=).""" + folder = tmp_path / "downloads" + pages[b"%PDF-1.7 silver"] = _pages(test_cost="$90") + _download(folder) + _download(folder, "Silver_70_SBC.pdf", b"%PDF-1.7 silver") + _download(folder, "bronze.pdf") # a crawlable link: never imported by hand + _download(folder, "unrelated.pdf") + (folder / "notes.txt").write_text("not a PDF") + + outcomes = import_folder(folder, YEAR) + + assert outcomes == { + "Silver_70_SBC.pdf": "ok", + "bronze.pdf": "not imported: no manual-only link ends in this file name", + "gold.pdf": "ok", + "unrelated.pdf": "not imported: no manual-only link ends in this file name", + } + assert _document(session, links["query"]).acquisition == "manual" + assert _document(session, links["crawl"]) is None + + +def test_a_file_name_two_links_share_must_be_imported_with_its_url(pages, links, session, tmp_path, sbc_hosts): + other = Plan(issuer_id=session.scalar(select(Issuer.id).where(Issuer.hios_issuer_id == "11111")), + hios_plan_id="11111CA0010009", plan_year=YEAR, marketing_name="Gold 80 HMO", metal_level="Gold", + plan_type="HMO", state="CA", benefits_url=f"{sbc_hosts['manual']}north/gold.pdf", + hsa_eligible=False, has_national_network=False, catalog_source=CATALOG_SOURCE) + session.add(other) + session.flush() + + outcomes = import_folder(_download(tmp_path / "downloads").parent, YEAR) + + assert outcomes == {"gold.pdf": "not imported: 2 links end in this file name; use FILE= and URL="} + + +def test_the_command_takes_one_file_with_its_url_or_a_folder(): + parsed = import_pdf.parse_args(["--year", "2026", "--file", "~/Downloads/x.pdf", "--url", "https://a.example/x.pdf"]) + assert parsed.file == Path.home() / "Downloads" / "x.pdf" # make passes the path quoted, so ~ is expanded here + assert import_pdf.parse_args(["--year", "2026", "--dir", "~/sbc"]).dir == Path.home() / "sbc" + + for args in (["--file", "x.pdf"], ["--dir", "d", "--url", "https://a.example/x.pdf"], + ["--file", "x.pdf", "--dir", "d"], []): + with pytest.raises(SystemExit): + import_pdf.parse_args(["--year", "2026", *args]) + + +def test_the_command_reports_a_refusal_without_a_traceback(pages, links, tmp_path): + with pytest.raises(SystemExit, match="Not imported: .* is not a PDF"): + import_pdf.main(["--year", str(YEAR), "--file", str(_download(tmp_path / "d", body=b"")), + "--url", links["manual"]]) + + +def test_the_command_prints_each_files_outcome(pages, links, tmp_path, capsys): + import_pdf.main(["--year", str(YEAR), "--dir", str(_download(tmp_path / "d").parent)]) + + assert capsys.readouterr().out == "gold.pdf: ok\n" + + +def test_the_command_says_when_a_folder_has_no_pdfs(pages, links, tmp_path): + (tmp_path / "empty").mkdir() + + with pytest.raises(SystemExit, match="No PDFs in"): + import_pdf.main(["--year", str(YEAR), "--dir", str(tmp_path / "empty")]) + + +def test_a_matched_file_that_fails_a_check_is_reported_and_the_rest_imported(pages, links, session, tmp_path): + folder = tmp_path / "downloads" + _download(folder) + _download(folder, "Silver_70_SBC.pdf", b"Your session has expired") + + outcomes = import_folder(folder, YEAR) + + assert outcomes["gold.pdf"] == "ok" + assert outcomes["Silver_70_SBC.pdf"] == "not imported: Silver_70_SBC.pdf is not a PDF" + assert _document(session, links["query"]) is None + + +def test_the_command_imports_one_file_and_prints_its_status(pages, links, tmp_path, capsys): + import_pdf.main(["--year", str(YEAR), "--file", str(_download(tmp_path / "d")), "--url", links["manual"]]) + + assert capsys.readouterr().out == "gold.pdf: ok\n" + + +def test_the_command_refuses_a_folder_that_is_not_one(tmp_path): + with pytest.raises(SystemExit, match="is not a folder"): + import_pdf.main(["--year", str(YEAR), "--dir", str(tmp_path / "nowhere")]) diff --git a/tests/test_sbc_ingest.py b/tests/test_sbc_ingest.py index b1a38bb..16a8144 100644 --- a/tests/test_sbc_ingest.py +++ b/tests/test_sbc_ingest.py @@ -93,7 +93,7 @@ def fetches(sbc, monkeypatch): @pytest.fixture def catalog(monkeypatch): - plan = ingest.PlanRef("99999NH0010001", "Example Gold", "Example Health") + plan = ingest.PlanRef("99999NH0010001", "Example Gold", "Example Health", "NH") monkeypatch.setattr(ingest, "documents_for", lambda session, states, year, issuer_ids: {GOLD: [plan], SILVER: [plan]}) @@ -381,3 +381,116 @@ def test_a_partly_read_document_is_neither_a_failure_nor_a_skipped_one(sbc, fetc assert ingest.execute(["NH"], YEAR) == Counter() assert "recorded failures skipped" not in capsys.readouterr().out assert fetches == [] + + +# California: links chosen by hand, read only as the source registry allows (ADR 0026) + +@pytest.fixture +def california(sbc, sbc_hosts, monkeypatch): + """One California link on each kind of host, served like the others; returns the links by host.""" + links = {kind: f"{prefix}gold.pdf" for kind, prefix in sbc_hosts.items() if kind != "texas"} + links["unregistered"] = "https://nobody.example.com/gold.pdf" + for url in links.values(): + sbc[url] = _pages() + plans = {url: [ingest.PlanRef(f"11111CA001000{n}", "Gold 80 HMO", "Example Care", "CA")] + for n, url in enumerate(links.values(), 1)} + monkeypatch.setattr(ingest, "documents_for", lambda session, states, year, issuer_ids: plans) + return links + + +def test_a_california_link_is_fetched_only_when_its_host_may_be_crawled(california, fetches, session, capsys): + assert ingest.execute(["CA"], YEAR) == Counter({"ok": 1}) + + assert fetches == [california["crawl"]] + out = capsys.readouterr().out + assert "4 SBC documents behind the CA plans for 1999; 0 already current, reading 1" in out + assert "1 awaiting a manual import (their host is manual-only): `make import-sbc YEAR=1999 DIR=`" in out + assert "2 links skipped: their host has no enabled sbc_host entry in the source registry" in out + assert _status(session, california["manual"]) is None + + +def test_an_api_states_links_are_read_without_the_registry(sbc, fetches, sbc_hosts, monkeypatch): + """Their links come from CMS; extending the gate to them is a follow-up (ADR 0026).""" + plan = ingest.PlanRef("99999NH0010001", "Example Gold", "Example Health", "NH") + monkeypatch.setattr(ingest, "documents_for", lambda session, states, year, issuer_ids: {GOLD: [plan]}) + + assert ingest.execute(["NH"], YEAR) == Counter({"ok": 1}) + assert fetches == [GOLD] + + +def _import_by_hand(url): + """What `make import-sbc` leaves: the file in the cache, parsed and marked manual.""" + ingest.cache_path(url, YEAR).write_bytes(url.encode()) + return ingest.ingest_document(url, YEAR, acquisition="manual") + + +def test_a_re_read_keeps_a_manual_document_marked_manual(california, session): + url = california["manual"] + assert _import_by_hand(url) == "ok" + + ingest.ingest_document(url, YEAR, _row(session, url)) + + assert session.scalar(select(SbcDocument.acquisition).where(SbcDocument.url == url)) == "manual" + + +def test_a_crawled_document_is_recorded_as_crawled(sbc, session): + ingest.ingest_document(GOLD, YEAR) + + assert session.scalar(select(SbcDocument.acquisition).where(SbcDocument.url == GOLD)) == "crawl" + + +def test_after_a_parser_change_a_manual_document_is_read_again_from_disk(california, fetches, session, + monkeypatch, tmp_path): + url = california["manual"] + _import_by_hand(url) + monkeypatch.setattr(ingest, "PARSER_VERSION", ingest.PARSER_VERSION + 1) + sbc_file = ingest.cache_path(url, YEAR) + before = sbc_file.stat().st_mtime_ns + fetches.clear() + + ingest.execute(["CA"], YEAR) + + assert url in fetches # served from the kept file, as fetch_pdf does + assert sbc_file.stat().st_mtime_ns == before # not downloaded again + assert _version(session, url) == ingest.PARSER_VERSION + assert session.scalar(select(SbcDocument.acquisition).where(SbcDocument.url == url)) == "manual" + + +def test_a_manual_document_whose_file_is_missing_is_never_fetched(california, fetches, session, capsys): + url = california["manual"] + _import_by_hand(url) + ingest.cache_path(url, YEAR).rename(ingest.cache_path(url, YEAR).with_suffix(".moved")) + fetches.clear() + + ingest.execute(["CA"], YEAR) + + assert url not in fetches + assert "1 manually imported PDFs are missing from" in capsys.readouterr().out + assert _status(session, url) == "ok" + + +def test_a_california_run_with_no_links_says_to_apply_the_manifest(sbc, monkeypatch): + monkeypatch.setattr(ingest, "documents_for", lambda session, states, year, issuer_ids: {}) + + with pytest.raises(SystemExit, match="make apply-sbc-manifest YEAR=1999"): + ingest.execute(["CA"], YEAR) + + +def test_california_is_a_state_the_ingest_takes_and_all_is_still_the_api_states(): + assert ingest.parse_args(["--states", "CA"]).states == ["CA"] + assert "CA" not in ingest.parse_args(["--states", "ALL"]).states + with pytest.raises(SystemExit): + ingest.parse_args(["--states", "NY"]) + + +def test_a_recorded_failure_is_counted_as_skipped_not_as_current(sbc, fetches, catalog, session, capsys): + """Counted as both, a blocked document read as healthy in the run's first line.""" + sbc[SILVER] = None + ingest.execute(["NH"], YEAR) + capsys.readouterr() + + ingest.execute(["NH"], YEAR) + + out = capsys.readouterr().out + assert "2 SBC documents behind the NH plans for 1999; 1 already current, reading 0" in out + assert "1 recorded failures skipped" in out diff --git a/tests/test_sbc_manifest.py b/tests/test_sbc_manifest.py new file mode 100644 index 0000000..e67394a --- /dev/null +++ b/tests/test_sbc_manifest.py @@ -0,0 +1,266 @@ +"""California's SBC links: a hand-built manifest, checked, then applied (ADR 0026). + +Real Postgres through the rolled-back `session` fixture, plan year 1999. The +carrier hosts are the made-up ones of the `sbc_hosts` fixture, except where +the committed manifest itself is checked against the committed registry. +""" +from contextlib import contextmanager +from datetime import date + +import pytest +from sqlalchemy import select + +from src.ingestion.ca_puf.load import CATALOG_SOURCE +from src.ingestion.sbc import manifest +from src.ingestion.sbc.manifest import ( + COLUMNS, + ManifestError, + apply_manifest, + check, + load_manifest, + name_matches, +) +from src.models.plan import Issuer, Plan +from src.models.sbc import SbcDocument + +YEAR = 1999 +GOLD, SILVER, BRONZE = "11111CA0010001", "11111CA0010002", "11111CA0010003" +HEADER = ",".join(COLUMNS) + + +def _write(tmp_path, *lines, header=HEADER): + path = tmp_path / f"ca-{YEAR}.csv" + path.write_text("\n".join([header, *lines]) + "\n", encoding="utf-8") + return path + + +def _line(plan, url, year=YEAR, verified="2026-09-26", note="checked"): + return f"{plan},{year},{url},{verified},{note}" + + +@pytest.fixture +def links(sbc_hosts): + """One link per host, by host id.""" + return {id_: f"{prefix}{id_}.pdf" for id_, prefix in sbc_hosts.items()} + + +@pytest.fixture +def plans(session): + """Three California plans loaded from the PUF, and one API-state plan with a link of its own.""" + issuer = Issuer(hios_issuer_id="11111", plan_year=YEAR, name="Example Care", state="CA") + texas = Issuer(hios_issuer_id="22222", plan_year=YEAR, name="Texas Health", state="TX") + session.add_all([issuer, texas]) + session.flush() + for owner, plan_id, name, state, source, url in [ + (issuer, GOLD, "Gold 80 HMO", "CA", CATALOG_SOURCE, None), + (issuer, SILVER, "Silver 70 HMO", "CA", CATALOG_SOURCE, None), + (issuer, BRONZE, "Bronze 60 HMO", "CA", CATALOG_SOURCE, None), + (texas, "22222TX0010001", "Texas Gold", "TX", "cms_api", "https://texas.example.com/sbc/own.pdf"), + ]: + session.add(Plan(issuer_id=owner.id, hios_plan_id=plan_id, plan_year=YEAR, marketing_name=name, + metal_level="Gold", plan_type="HMO", state=state, benefits_url=url, + hsa_eligible=False, has_national_network=False, catalog_source=source)) + session.flush() + + +def _links(session): + return dict(session.execute(select(Plan.hios_plan_id, Plan.benefits_url).where(Plan.plan_year == YEAR)).all()) + + +def _rows(tmp_path, *lines): + return load_manifest(_write(tmp_path, *lines), YEAR) + + +# Loading: the file on its own, with no database + +def test_a_valid_manifest_is_read_into_rows(tmp_path, links): + rows = _rows(tmp_path, _line(GOLD, links["crawl"]), _line(SILVER, links["manual"], note=" ")) + + assert rows == (manifest.ManifestRow(GOLD, links["crawl"], date(2026, 9, 26), "checked"), + manifest.ManifestRow(SILVER, links["manual"], date(2026, 9, 26), "")) + + +@pytest.mark.parametrize(("line", "message"), [ + (_line("11111CA001000", "{crawl}"), "is not a California base plan ID"), + (_line("11111CA0010001-01", "{crawl}"), "is not a California base plan ID"), + (_line("11111TX0010001", "{crawl}"), "is not a California base plan ID"), + (_line(GOLD, "{crawl}", year=YEAR + 1), f"plan year '{YEAR + 1}' is not {YEAR}"), + (_line(GOLD, "http://crawl.example.com/sbc/gold.pdf"), "the link is"), + (_line(GOLD, "https://127.0.0.1/sbc/gold.pdf"), "the link is"), + (_line(GOLD, "https://nobody.example.com/gold.pdf"), "no California sbc_host entry"), + (_line(GOLD, "{texas}"), "no California sbc_host entry"), + (_line(GOLD, "{crawl}", verified="26/09/2026"), "verified_on must be a date"), + (_line(GOLD, "{crawl}") + ",extra", "expected 5 columns"), + (f"{GOLD},{YEAR},{{crawl}}", "expected 5 columns"), +]) +def test_a_bad_line_is_named_and_nothing_is_read(tmp_path, links, line, message): + with pytest.raises(ManifestError, match=message) as error: + _rows(tmp_path, _line(SILVER, links["crawl"]), line.format(**links)) + + assert "line 3" in str(error.value) + + +def test_the_header_must_be_exactly_the_columns(tmp_path, links): + with pytest.raises(ManifestError, match="the header must be"): + load_manifest(_write(tmp_path, _line(GOLD, links["crawl"]), header="plan,year,url,verified,note"), YEAR) + + +def test_a_plan_listed_twice_is_refused(tmp_path, links): + with pytest.raises(ManifestError, match=f"plan {GOLD} is listed more than once"): + _rows(tmp_path, _line(GOLD, links["crawl"]), _line(GOLD, links["manual"])) + + +def test_a_file_that_is_not_utf8_is_a_manifest_error(tmp_path): + path = tmp_path / "ca.csv" + path.write_bytes(b"\xff\xfe" + HEADER.encode("utf-16-le")) + + with pytest.raises(ManifestError, match="could not be read"): + load_manifest(path, YEAR) + + +def test_the_committed_2026_manifest_is_valid_against_the_committed_registry(): + """The file a person edits by hand; a bad row must fail here, not in a yearly run.""" + rows = load_manifest(manifest.manifest_path(2026), 2026) + + assert len(rows) == 114 + assert {row.hios_plan_id[:5] for row in rows} == { + "18126", "27603", "40513", "51396", "70285", "84014", "92499", "92815", "93689"} + + +# Applying: the manifest decides every California plan's link + +def test_applying_links_the_listed_plans_and_leaves_the_rest_with_none(tmp_path, session, plans, links): + report = apply_manifest(session, _rows(tmp_path, _line(GOLD, links["crawl"]), _line(SILVER, links["crawl"])), + YEAR) + + assert _links(session) == {GOLD: links["crawl"], SILVER: links["crawl"], BRONZE: None, + "22222TX0010001": "https://texas.example.com/sbc/own.pdf"} + assert report == manifest.ApplyReport(linked=2, documents=1, unlisted=(BRONZE,), disabled={}) + + +def test_a_row_taken_out_of_the_manifest_takes_its_link_away(tmp_path, session, plans, links): + apply_manifest(session, _rows(tmp_path, _line(GOLD, links["crawl"]), _line(SILVER, links["manual"])), YEAR) + + apply_manifest(session, _rows(tmp_path, _line(GOLD, links["crawl"])), YEAR) + + assert (_links(session)[GOLD], _links(session)[SILVER]) == (links["crawl"], None) + + +def test_a_disabled_carrier_is_unlinked_and_counted(tmp_path, session, plans, links): + """Disabling its registry entry and applying again is how a carrier is removed.""" + report = apply_manifest(session, _rows(tmp_path, _line(GOLD, links["crawl"]), _line(SILVER, links["off"])), YEAR) + + assert _links(session)[SILVER] is None + assert (report.linked, report.disabled) == (1, {"off": 1}) + + +def test_a_plan_the_catalog_does_not_have_stops_the_apply(tmp_path, session, plans, links): + before = _links(session) + + with pytest.raises(ManifestError, match="11111CA0010009 is in the manifest but not among"): + apply_manifest(session, _rows(tmp_path, _line(GOLD, links["crawl"]), _line("11111CA0010009", links["crawl"])), + YEAR) + + assert _links(session) == before + + +def test_a_missing_manifest_applies_nothing(monkeypatch, tmp_path): + monkeypatch.setattr(manifest, "MANIFEST_DIR", tmp_path) + + assert manifest.apply_for_year(YEAR) is None + + +# Checking: what a person should look at after an ingest + +@pytest.mark.parametrize(("name", "title", "matches"), [ + ("Silver 70 HMO", "Covered CA_Silver 70 HMO", True), + ("Bronze 60 HDHP Premier HMO", "Sharp Health Plan: Sharp Bronze 60 Premier HDHP HMO", True), + ("Gold 80 HMO", "Example Care: Gold 80 PPO", False), + ("Gold 80 HMO", None, False), +]) +def test_a_title_matches_when_it_prints_every_word_of_the_plans_name(name, title, matches): + assert name_matches(name, title) is matches + + +def test_the_check_sorts_every_plan_into_what_it_needs(tmp_path, session, plans, links): + issuer_id = session.scalar(select(Issuer.id).where(Issuer.hios_issuer_id == "11111")) + for plan_id, name in (("11111CA0010004", "Platinum 90 HMO"), ("11111CA0010005", "Minimum Coverage HMO")): + session.add(Plan(issuer_id=issuer_id, hios_plan_id=plan_id, plan_year=YEAR, marketing_name=name, + metal_level="Gold", plan_type="HMO", state="CA", hsa_eligible=False, + has_national_network=False, catalog_source=CATALOG_SOURCE)) + session.flush() + wrong, blocked = f"{links['crawl']}?wrong", f"{links['crawl']}?blocked" + apply_manifest(session, _rows(tmp_path, _line(GOLD, links["manual"]), _line(SILVER, wrong), + _line(BRONZE, links["crawl"]), _line("11111CA0010004", blocked)), YEAR) + session.add_all([ + SbcDocument(url=links["manual"], plan_year=YEAR, status="ok", title="Example Care: Gold 80 HMO", + acquisition="manual"), + SbcDocument(url=wrong, plan_year=YEAR, status="ok", title="Example Care: Gold 80 HMO"), + SbcDocument(url=blocked, plan_year=YEAR, status="blocked"), + ]) + session.flush() + + report = check(session, YEAR) + + assert (report.read, report.manual) == (2, 1) + assert report.name_mismatch == [("Example Care", SILVER, "Silver 70 HMO", "Example Care: Gold 80 HMO")] + assert report.failed == [("Example Care", "11111CA0010004", "Platinum 90 HMO", "blocked")] + assert report.not_read == [("Example Care", BRONZE, "Bronze 60 HMO")] + assert report.no_link == [("Example Care", "11111CA0010005", "Minimum Coverage HMO")] + assert report.awaiting_import == [] + + +def test_an_unread_link_on_a_manual_host_is_awaiting_import(tmp_path, session, plans, links): + apply_manifest(session, _rows(tmp_path, _line(GOLD, links["manual"])), YEAR) + + report = check(session, YEAR) + + assert report.awaiting_import == [("Example Care", GOLD, "Gold 80 HMO")] + assert report.not_read == [] + + +# The command + +@pytest.fixture +def command(monkeypatch, session, tmp_path): + @contextmanager + def same_session(): + yield session + session.flush() + + monkeypatch.setattr(manifest, "get_session", same_session) + monkeypatch.setattr(manifest, "MANIFEST_DIR", tmp_path) + + +def test_the_apply_command_says_what_it_linked(command, tmp_path, session, plans, links, capsys): + _write(tmp_path, _line(GOLD, links["crawl"]), _line(SILVER, links["off"])) + + manifest.main(["apply", "--year", str(YEAR)]) + + out = capsys.readouterr().out + assert f"1 California plans for {YEAR} linked to 1 SBC documents; 1 not in the manifest" in out + assert "1 plans unlinked: registry entry off is disabled" in out + + +def test_the_apply_command_exits_with_the_broken_rule(command, tmp_path, plans, links): + _write(tmp_path, _line(GOLD, "https://nobody.example.com/gold.pdf")) + + with pytest.raises(SystemExit, match="SBC manifest not applied: line 2"): + manifest.main(["apply", "--year", str(YEAR)]) + + +def test_the_apply_command_names_the_file_it_expected(command, tmp_path): + with pytest.raises(SystemExit, match=f"No manifest for {YEAR}: expected .*ca-{YEAR}.csv"): + manifest.main(["apply", "--year", str(YEAR)]) + + +def test_the_check_command_prints_each_list_with_its_heading(command, tmp_path, session, plans, links, capsys): + _write(tmp_path, _line(GOLD, links["manual"])) + manifest.main(["apply", "--year", str(YEAR)]) + + manifest.main(["check", "--year", str(YEAR)]) + + out = capsys.readouterr().out + assert f"California SBCs for {YEAR}: 0 plans read (0 from manual imports)" in out + assert "Awaiting `make import-sbc`" in out and f"Example Care | {GOLD} | Gold 80 HMO" in out + assert "No link in the manifest: 2" in out diff --git a/tests/test_sbc_refresh.py b/tests/test_sbc_refresh.py index 1697502..363af0f 100644 --- a/tests/test_sbc_refresh.py +++ b/tests/test_sbc_refresh.py @@ -217,7 +217,7 @@ def test_a_document_that_is_gone_or_now_refused_loses_its_text_but_not_its_file( def test_a_run_reads_what_is_new_and_leaves_recorded_failures_to_a_refresh(issuer, session, catalog, monkeypatch, tmp_path): monkeypatch.setattr(ingest, "documents_for", - lambda session, states, year, issuer_ids: {GOLD: [ingest.PlanRef("99999NH0010001", "Gold", "Example")]}) + lambda session, states, year, issuer_ids: {GOLD: [ingest.PlanRef("99999NH0010001", "Gold", "Example", "NH")]}) issuer.reply = FetchResult("blocked", detail="robots.txt disallows it") assert ingest.execute(["NH"], YEAR) == Counter({"blocked": 1}) @@ -229,7 +229,7 @@ def test_a_run_reads_what_is_new_and_leaves_recorded_failures_to_a_refresh(issue def test_a_refresh_asks_about_every_stored_document(issuer, session, catalog, monkeypatch, tmp_path): monkeypatch.setattr(ingest, "documents_for", - lambda session, states, year, issuer_ids: {GOLD: [ingest.PlanRef("99999NH0010001", "Gold", "Example")]}) + lambda session, states, year, issuer_ids: {GOLD: [ingest.PlanRef("99999NH0010001", "Gold", "Example", "NH")]}) ingest.execute(["NH"], YEAR) issuer.reply = FetchResult(NOT_MODIFIED) @@ -257,3 +257,15 @@ def test_every_pdf_ever_written_is_still_on_disk_after_any_run(issuer, session, def test_refresh_is_a_flag_on_the_ingest_command(): assert ingest.parse_args(["--states", "NH", "--refresh"]).refresh is True assert ingest.parse_args(["--states", "NH"]).refresh is False + + +def test_a_refresh_never_asks_the_carrier_about_a_document_imported_by_hand(issuer, session, catalog, monkeypatch, + tmp_path): + """A person downloaded it; asking the carrier again is what the manual route exists to avoid (ADR 0026).""" + monkeypatch.setattr(ingest, "documents_for", + lambda session, states, year, issuer_ids: {GOLD: [ingest.PlanRef("99999NH0010001", "Gold", "Example", "NH")]}) + ingest.ingest_document(GOLD, YEAR, acquisition="manual") + + assert ingest.execute(["NH"], YEAR, refresh=True) == Counter() + assert issuer.asked == [] + assert _document(session).acquisition == "manual" diff --git a/tests/test_sbc_report.py b/tests/test_sbc_report.py index aa42cc3..1bc58f5 100644 --- a/tests/test_sbc_report.py +++ b/tests/test_sbc_report.py @@ -180,3 +180,19 @@ def test_verifying_files_checks_a_partly_read_documents_pdf_too(session, catalog assert report.missing_files == ["https://sbc.example.com/partial.pdf"] assert report.changed_files == [] + + +def test_read_documents_are_counted_by_how_they_were_acquired(session, catalog): + """A crawled document and one a person imported by hand (ADR 0026).""" + session.execute(SbcDocument.__table__.update() + .where(SbcDocument.url == "https://sbc.example.com/partial.pdf").values(acquisition="manual")) + + report = collect(session, YEAR) + + assert report.acquired == {"crawl": 1, "manual": 1} + assert "Documents read: 1 crawled, 1 imported by hand" in render(report) + + +def test_the_report_takes_california_and_all_is_still_the_api_states(): + assert sbc_report.parse_args(["--states", "CA"]).states == ["CA"] + assert "CA" not in sbc_report.parse_args(["--states", "ALL"]).states diff --git a/tests/test_tools.py b/tests/test_tools.py index 8d64c7f..7175a38 100644 --- a/tests/test_tools.py +++ b/tests/test_tools.py @@ -233,6 +233,37 @@ def test_coverage_returns_each_plans_passages_as_data_and_as_citations(): assert outcome.chunks == (passage,) +def _coverage_outcome(*coverages): + with patch("src.services.tools.coverage_for", return_value=list(coverages)), patch("src.services.tools.get_session"): + return run_tool("plan_coverage", _coverage_args([c.plan_id for c in coverages])) + + +def test_quoting_a_california_plans_sbc_opens_the_answer_with_the_standard_version_notice(): + """Server-written (ADR 0026): asked to repeat a note, the model left it out.""" + passage = RetrievedChunk("c1", "If you need immediate medical attention\nUrgent care $50", "Silver 70 - x.pdf", 0.4) + outcome = _coverage_outcome( + PlanCoverage("11111CA0010001", "ok", "Silver 70 HMO", "Example Care", 2026, "https://a.example/s.pdf", + (passage,), "ok", "CA"), + PlanCoverage("12345NH0010001", "ok", "Gold", "Example", 2026, "https://a.example/g.pdf", (passage,), "ok", "NH")) + + assert outcome.notice == ( + "Summaries of Benefits and Coverage quoted here for Covered California plans are for each plan's " + "standard version; people who qualify for cost-sharing reductions or American Indian and Alaska " + "Native cost sharing pay less than they show.") + assert "note" not in json.loads(outcome.content)["plans"][0] + + +@pytest.mark.parametrize("coverage", [ + PlanCoverage("12345NH0010001", "ok", "Gold", "Example", 2026, "https://a.example/g.pdf", + (RetrievedChunk("c1", "text", "Gold - x.pdf", 0.4),), "ok", "NH"), + PlanCoverage("11111CA0010001", "no_document", "Silver 70 HMO", "Example Care", 2026, "https://a.example/s.pdf", + sbc_status="not_read", state="CA"), + PlanCoverage("11111CA0010002", "not_found"), +]) +def test_no_notice_when_no_california_sbc_is_quoted(coverage): + assert _coverage_outcome(coverage).notice is None + + def _search_payload(result, zip_code="90012"): with patch("src.services.tools.search_plans", return_value=result), patch("src.services.tools.get_session"): return json.loads(run_tool("search_plans", _args(zip_code=zip_code, age=40)).content)