California SBCs: a hand-built manifest, Terms-gated crawling, and manual import

CHANGELOG.md

@@ -4,6 +4,72 @@
 
 ### Added
 
+- Design and ADR 0026 for California's SBCs, sub-project 2
+  (`docs/superpowers/specs/2026-09-25-ca-sbc-design.md`). CMS's California
+  file has no SBC links, so each plan's link comes from a hand-built manifest.
+  **Website Terms govern:** a carrier whose Terms ban robots or scrapers is
+  manual-only, even where `robots.txt` allows us, as Covered California already
+  was. ADR 0026 amends ADRs 0013, 0019 and 0025.
+- The source registry records the 11 California carriers as `sbc_host`
+  entries, each with its Terms, `robots.txt` outcome, access and removal
+  steps:
+  - **crawled:** IEHP, Molina, Sharp and Balance by CCHP;
+  - **manual-only:** Kaiser, Blue Shield, Western Health Advantage, Anthem,
+    L.A. Care and Valley;
+  - **disabled:** Health Net, whose SBCs sit behind a search form.
+
+  They use a new status, `mandated_disclosure`: an SBC is published because
+  federal law requires it, which is not a licence. So it is allowed only for an
+  `sbc_host`, and its commercial use stays `review`. `sbc_host_for` finds a
+  link's entry by the longest matching prefix, and every scope URL must now be
+  an `https://host/…` prefix, so it can't also match a look-alike host.
+- California's SBCs are linked from a committed, hand-built manifest,
+  `src/ingestion/sbc/manifests/ca-2026.csv`: 114 of 190 plans, 58 documents.
+  - `make apply-sbc-manifest YEAR=` checks every row first: plan ID, year, a
+    safe HTTPS link, and a California registry entry covering it. Then it sets
+    each California plan's link, and a plan with no row, or whose carrier is
+    disabled, gets none. `make ingest-ca-plans` applies it after each load.
+  - `make check-sbc-manifest YEAR=` lists, for a person to check:
+    - documents whose printed title lacks the plan's name;
+    - failures;
+    - plans awaiting a manual import;
+    - unread links;
+    - plans with no link.
+- `make ingest-sbc STATES=CA` and `make sbc-report STATES=CA`. California links
+  are read only as their carrier's entry allows:
+  - `crawl` is fetched;
+  - `manual` is never requested, and is counted as awaiting import;
+  - disabled or unregistered is skipped.
+
+  `STATES=ALL` is still the 30 HealthCare.gov states. First run: the 19 IEHP,
+  Molina and Sharp SBCs all read `ok`, with no name mismatches.
+- `make import-sbc YEAR= DIR=` (or `FILE= URL=`) reads an SBC a person
+  downloaded in a browser.
+  - **Checks:** the link must be a plan's and its carrier enabled, and the file
+    a PDF within the size cap.
+  - **What it does:** it is copied to the link's cache path, and any different
+    file there is archived. It is then parsed like a crawled one, with no
+    request. `DIR=` matches each file to the one manual-only link with its
+    name, or its `?fileName=` value.
+  - **Afterwards:** the document is recorded as `manual` and never re-requested.
+    `refresh-sbc` skips it; after a parser change it is re-parsed from disk,
+    and a missing file is reported for re-import.
+- Migration `3b1d6e2a9c47`: `sbc_documents.acquisition`, `crawl` or `manual`,
+  checked, default `crawl`. `sbc-report` counts read documents by it.
+- California coverage answers say that a plan's SBC is its standard version,
+  so people who qualify for cost-sharing reductions or American Indian and
+  Alaska Native cost sharing pay less than it shows. The server writes the
+  notice, like the prior-year one. A first version asked the model to repeat
+  a note from the tool result, and in a live question it left it out.
+- `docs/findings/ca-sbc.md`: where each California carrier publishes its SBCs,
+  what its Terms and `robots.txt` say (quoted), how each manifest row was
+  checked, what scripts requested before the Terms were read, and the first
+  run's counts. The eval still passes: 1,303 of 1,320 ranking questions put
+  the right section in the top 4 (98.7%; floor 97%).
+- Runbooks: `california.md` gains the yearly SBC routine (re-check Terms, write
+  the manifest, apply, crawl, import, check), and `sbc.md` how manual documents
+  behave in the monthly one. The README documents the new commands and the
+  registry's `mandated_disclosure`.
 - **PolicyPal is deployed to Azure.** The API runs on Container Apps in
   Central US on the image digest built from `main`, backed by a PostgreSQL 16
   Flexible Server with `pgvector`, and the browser app is on Static Web Apps.
@@ -447,6 +513,9 @@
 
 ### Fixed
 
+- `make ingest-sbc` counted a recorded failure twice: once as "already
+  current" and again as a "recorded failure skipped". So a run over blocked
+  documents read as healthier than it was. It now counts it once, as skipped.
 - "Compare the gold plans." and "Compare silver plans for me." skipped the plan
   search in about half of runs, in every state. The prompt left it open whether
   comparing plans of a metal level meant real plans or the level in general,

Makefile

@@ -1,4 +1,5 @@
-.PHONY: ingest build-index ingest-plans ingest-ca-plans ingest-sbc refresh-sbc sbc-report migrate api \
+.PHONY: ingest build-index ingest-plans ingest-ca-plans ingest-sbc refresh-sbc sbc-report apply-sbc-manifest \
+        check-sbc-manifest import-sbc migrate api \
         ui-dev ui-build ui-lint ui-preview ui-test test lint check
 
 # The plan year to work on; the commands default to the calendar year, which
@@ -29,7 +30,8 @@ ingest-ca-plans:
 	@test -n "$(YEAR)" || { echo "YEAR is required, e.g. make ingest-ca-plans YEAR=2026"; exit 2; }
 	uv run python -m src.ingestion.ca_puf --year $(YEAR) $(if $(ZIP),--zip $(ZIP)) $(if $(REFRESH),--refresh)
 
-# Summary of Benefits PDFs for the catalog plans in STATES; run ingest-plans
+# Summary of Benefits PDFs for the catalog plans in STATES; run ingest-plans (for
+# CA, ingest-ca-plans and its SBC manifest: ADR 0026)
 # first. Required for the same reason. e.g. make ingest-sbc STATES=NH,DE
 # TOP_ISSUERS=1 reads only the largest parent companies' plans, ISSUERS=40788,66252
 # only those HIOS issuers' (ADR 0015). Downloaded PDFs are always kept (ADR 0016).
@@ -49,6 +51,27 @@ refresh-sbc:
 sbc-report:
 	uv run python -m src.ingestion.sbc.report $(YEAR_ARG) $(if $(STATES),--states $(STATES)) $(if $(VERIFY),--verify-files)
 
+# California's SBC links, from the committed src/ingestion/sbc/manifests/ca-YEAR.csv
+# (ADR 0026). ingest-ca-plans applies it too; run this after editing the file
+# or the source registry. e.g. make apply-sbc-manifest YEAR=2026
+apply-sbc-manifest:
+	@test -n "$(YEAR)" || { echo "YEAR is required, e.g. make apply-sbc-manifest YEAR=2026"; exit 2; }
+	uv run python -m src.ingestion.sbc.manifest apply --year $(YEAR)
+
+# After `make ingest-sbc STATES=CA`: documents whose printed title doesn't
+# name their plan, failures, and what still awaits a manual import. Reads only.
+check-sbc-manifest:
+	@test -n "$(YEAR)" || { echo "YEAR is required, e.g. make check-sbc-manifest YEAR=2026"; exit 2; }
+	uv run python -m src.ingestion.sbc.manifest check --year $(YEAR)
+
+# An SBC a person downloaded in a browser, for a carrier whose documents are
+# manual-only (ADR 0026). FILE=x.pdf URL=<its link in the manifest>, or DIR=folder
+# to match files to links by file name. e.g. make import-sbc YEAR=2026 DIR=~/Downloads/ca-sbc
+import-sbc:
+	@test -n "$(YEAR)" || { echo "YEAR is required, e.g. make import-sbc YEAR=2026 DIR=folder"; exit 2; }
+	@test -n "$(FILE)$(DIR)" || { echo "FILE= with URL=, or DIR=, is required"; exit 2; }
+	uv run python -m src.ingestion.sbc.import_pdf --year $(YEAR) $(if $(FILE),--file "$(FILE)" --url "$(URL)") $(if $(DIR),--dir "$(DIR)")
+
 migrate:
 	uv run alembic upgrade head
 

README.md

@@ -275,7 +275,9 @@ Every source PolicyPal reads or links is recorded in
 [`src/ingestion/sources/registry.toml`](src/ingestion/sources/registry.toml),
 with its licence, permission status, `robots.txt` status and removal steps
 (ADR 0025). Only approved sources run. Covered California is recorded as
-link-only: its Terms of Use forbid automated access.
+link-only: its Terms of Use forbid automated access. Each California carrier
+whose SBCs are used is an `sbc_host` entry with status `mandated_disclosure`,
+which is not a licence, so its commercial use stays under review (ADR 0026).
 Verified API behaviour is recorded in
 [`docs/findings/cms-marketplace-api.md`](docs/findings/cms-marketplace-api.md).
 
@@ -318,6 +320,30 @@ sections into `sbc_chunks`, kept apart from the corpus that `make ingest` rebuil
   (ADR 0019), so ingesting another state does not re-request every blocked host.
 - For a quick check: `uv run python -m src.ingestion.sbc --states NH --limit 5`.
 
+#### California's SBCs
+
+```bash
+make apply-sbc-manifest YEAR=2026            # links from src/ingestion/sbc/manifests/ca-2026.csv
+make ingest-sbc STATES=CA YEAR=2026          # reads only the carriers that may be crawled
+make import-sbc YEAR=2026 DIR=~/Downloads/ca-sbc   # PDFs a person downloaded in a browser
+make check-sbc-manifest YEAR=2026            # what still needs a person's eye
+```
+
+CMS's California file has no SBC links. A committed, hand-built manifest gives
+each plan its link, and each carrier has a registry entry (ADR 0026).
+
+- **Crawled:** carriers whose `robots.txt` and website Terms both allow
+  automated access.
+- **Manual only:** carriers whose Terms ban robots or scrapers, and hosts that
+  block bots. A person downloads their SBCs in a browser, and `make import-sbc`
+  reads the file with no request to the carrier. The document is recorded as
+  `manual` and never re-requested.
+
+California answers say the SBC is the plan's standard version, since
+cost-sharing reductions lower what it shows. The yearly steps are in
+[docs/runbooks/california.md](docs/runbooks/california.md), and the carrier
+research in [docs/findings/ca-sbc.md](docs/findings/ca-sbc.md).
+
 ### Keeping documents current
 
 ```bash

docs/decisions/0013-sbc-documents.md

@@ -10,6 +10,8 @@ Amended by ADR 0018: chart rows are rebuilt from the table's ruled grid, one
 line per service, and a PDF with no text layer is refused as scanned.
 Its "a refused `robots.txt` counts as disallowing everything" is **superseded
 by ADR 0020**, which follows RFC 9309's reading of an unavailable robots.txt.
+Amended by ADR 0026 for California: a carrier's website Terms can refuse us
+too, and a document a person downloads in a browser may be imported by hand.
 
 ## Context
 

docs/decisions/0019-keeping-sbcs-current.md

@@ -4,7 +4,8 @@
 
 Accepted. Phase 4. Supersedes ADR 0015's "Known limits, left to Phase 4", and
 amends ADR 0013's "a failure is retried on the next run" and ADR 0016's
-folders.
+folders. Amended by ADR 0026: a document imported by hand is never
+re-requested, by `ingest-sbc` or `refresh-sbc`.
 
 ## Context
 

docs/decisions/0025-a-committed-source-registry.md

@@ -3,7 +3,9 @@
 ## Status
 
 Accepted. Records in data what ADR 0003 (corpus sources) and ADR 0013 (SBC
-copyright) decided in prose.
+copyright) decided in prose. Amended by ADR 0026: a fourth approved status,
+`mandated_disclosure`, for `sbc_host` entries only, and every scope URL must
+be an `https://host/…` prefix.
 
 ## Context
 
@@ -28,7 +30,7 @@ new dependency:
 | --- | --- |
 | `kind` | `corpus`, `catalog`, `sbc_host`, `directory` |
 | `jurisdiction` | `US` or a state code |
-| `permission_status` | `public_domain`, `open_license`, `written_permission`, `pending_review`, `denied` |
+| `permission_status` | `public_domain`, `open_license`, `written_permission`, `mandated_disclosure` (ADR 0026: `sbc_host` only, `commercial_use = "review"`), `pending_review`, `denied` |
 | `commercial_use` | `yes`, `no`, `review` |
 | `robots` | `allowed`, `disallowed`, `unavailable`, `not_applicable`; `robots_checked_on` is required unless `not_applicable` |
 | `access` | `api`, `download`, `crawl`, `manual`, `link` |
@@ -40,7 +42,10 @@ Every entry also has `id`, `name`, `publisher`, `scope_urls`, `license`,
 - required fields are present, and no unknown field;
 - values are from the lists above;
 - ids are unique;
-- `enabled = true` needs `public_domain`, `open_license` or `written_permission`.
+- `enabled = true` needs `public_domain`, `open_license`, `written_permission`
+  or, since ADR 0026, `mandated_disclosure`;
+- each `scope_urls` entry is an `https://host/…` prefix (ADR 0026), so it
+  cannot also match a look-alike host.
 
 An invalid registry fails the import, so a bad edit cannot quietly enable
 anything.