Read an unavailable robots.txt as the standard does, not as a refusal

CHANGELOG.md

@@ -201,6 +201,17 @@
 
 ### Changed
 
+- A `robots.txt` the server cannot serve is no longer read as a refusal
+  (ADR 0020, superseding ADR 0013). RFC 9309 puts every 4xx in one
+  "unavailable" class where a crawler may fetch, and probing all seven
+  affected hosts found five serving their PDF at 200 `application/pdf`. A 5xx
+  now disallows, which it did not before. A real `Disallow` rule, and a 401 or
+  403 on the document itself, are still refusals and still never worked
+  around. Refreshing the eight affected states took coverage from 1,940 plans
+  (59.2%) to **2,274 (69.4%)**. Those documents parse less completely than the
+  ones already read, so `partial` went from 22 to 138: the figure is 2,136
+  whole documents and 138 partial, not 2,274 whole. No PDF was deleted; the
+  archive grew by the 77 files issuers had replaced.
 - The COVERAGE eval floor is full marks, 10 of 10, raised from 9. Parser
   version 3 reads CHRISTUS's wrapped imaging row as one line, so the answer
   states its price in 6 tries of 6, and the set ran 10/10 in three runs.

README.md

@@ -4,7 +4,7 @@
 
 PolicyPal lets a logged-in user ask natural-language questions about any kind of insurance — health, auto, life, home, and more — and returns grounded answers generated by an LLM using context retrieved from a vector store. Every answer cites the source passages it was built from.
 
-**Built:** comparison of real purchasable health plans drawn from the CMS Marketplace API. That half is deliberately narrower than the Q&A — ACA individual and family plans, in the 30 states served by HealthCare.gov. The 21 state-based exchanges, employer coverage, Medicare and Medicaid have no equivalent public catalog and are out of scope. All five phases of [docs/plans](docs/plans/) are complete; the catalog is loaded for eighteen states (3,276 plans, 137 issuers), and 1,940 of those plans have their Summary of Benefits read here. Keeping the documents current is a routine now, not a phase: [docs/runbooks/sbc.md](docs/runbooks/sbc.md).
+**Built:** comparison of real purchasable health plans drawn from the CMS Marketplace API. That half is deliberately narrower than the Q&A — ACA individual and family plans, in the 30 states served by HealthCare.gov. The 21 state-based exchanges, employer coverage, Medicare and Medicaid have no equivalent public catalog and are out of scope. All five phases of [docs/plans](docs/plans/) are complete; the catalog is loaded for eighteen states (3,276 plans, 137 issuers), and 2,274 of those plans (69.4%) have their Summary of Benefits read here — 2,136 read whole and 138 in part (ADR 0017). Keeping the documents current is a routine now, not a phase: [docs/runbooks/sbc.md](docs/runbooks/sbc.md).
 
 PolicyPal will compare plans; it will not recommend one. Advice-shaped questions are refused by design, and that refusal is a measured property with its own regression floor (ADR 0004).
 
@@ -303,7 +303,7 @@ yields no table, or a questions column that interleaves its answers. Those are
 stored as `partial`: the text that was read is kept and searched, the plan card
 says the chart is not all there, and an answer that finds nothing says the part
 that would answer wasn't among what was read, rather than implying the plan
-doesn't cover it. 22 of the 1,246 documents stored are partial.
+doesn't cover it. 138 of the 1,498 documents stored are partial.
 
 ## Testing
 

docs/decisions/0013-sbc-documents.md

@@ -8,6 +8,8 @@ not on every run. ADR 0015 also deleted each PDF once its text was stored;
 ADR 0016 reverses that, and every downloaded SBC is kept.
 Amended by ADR 0018: chart rows are rebuilt from the table's ruled grid, one
 line per service, and a PDF with no text layer is refused as scanned.
+Its "a refused `robots.txt` counts as disallowing everything" is **superseded
+by ADR 0020**, which follows RFC 9309's reading of an unavailable robots.txt.
 
 ## Context
 
@@ -61,8 +63,11 @@ among the catalog plans of those states.
   every redirect, which is followed by hand rather than by `requests`.
 - **Refusals are respected.**
   - `robots.txt` is read and obeyed, using the standard library's parser.
-  - A 401 or 403, on the file or on `robots.txt`, records the document as
-    `blocked`. A refused `robots.txt` counts as disallowing everything.
+  - A 401 or 403 on the file records the document as `blocked`.
+  - ~~A refused `robots.txt` counts as disallowing everything.~~ **Superseded
+    by ADR 0020:** RFC 9309 puts 4xx in one "unavailable" class where a
+    crawler may fetch, and measuring all seven affected hosts found five
+    serving the document at 200. A 5xx now disallows, which this did not.
   - Nothing tries to get around a refusal: no browser user agent, no other
     route to the file. Those plans get no SBC answers; answers link their PDF
     instead.

docs/decisions/0020-an-unavailable-robots-txt-is-not-a-refusal.md

@@ -0,0 +1,123 @@
+# 0020 — An unavailable robots.txt is not a refusal
+
+## Status
+
+Accepted. Supersedes ADR 0013's "A refused `robots.txt` counts as disallowing
+everything."
+
+## Context
+
+ADR 0013 read `robots.txt` before every SBC fetch and recorded a site that
+refused it as `blocked`:
+
+> A 401 or 403, on the file or on `robots.txt`, records the document as
+> `blocked`. A refused `robots.txt` counts as disallowing everything.
+
+That was written from two hosts in a two-state slice. At eighteen states it
+governs **482 documents** — the second largest reason for missing coverage
+after a real `Disallow` rule.
+
+It is also stricter than the standard it implements. RFC 9309 §2.3.1.3 puts
+every 400–499 status in one class, "Unavailable", and says a crawler **may**
+access the site's resources; only §2.3.1.4's "Unreachable" (5xx) requires
+assuming a complete disallow. A 401 or 403 on `/robots.txt` is a web server
+declining to serve one file. It is not a statement about the documents.
+
+The costs chart of this decision was never measured, so it was measured. All
+482 documents come from **seven hosts**, so this is a census, not a sample:
+one request each, two seconds apart, status and content type only, nothing
+stored or parsed.
+
+| Host | Documents | `/robots.txt` | The PDF itself |
+| --- | --- | --- | --- |
+| www.uhc.com | 229 | 403 | **403** |
+| portal.medica.com | 117 | 403 | 200 `application/pdf` |
+| sbc.anthem.com | 76 | 401 | 200 `application/pdf` |
+| sbc.wellpoint.com | 34 | 401 | 200 `application/pdf` |
+| apps.avmed.com | 14 | 401 | 200 `application/pdf` |
+| www.mercyhealthsystem.org | 6 | 403 | **403** |
+| static.mending.com | 6 | 403 | 200 `application/pdf` |
+
+**Five of seven serve the document without objection.** The prediction going
+in was the opposite — that a 403 on `robots.txt` meant a WAF refusing the
+client outright, so the PDF would refuse too. That holds for exactly two
+hosts, and they are already refusing on their own account.
+
+## Decision
+
+### The status of `robots.txt` decides, as RFC 9309 says
+
+- **2xx** — these are the rules, parsed and obeyed. Unchanged: a real
+  `Disallow` is still a refusal, and 483 documents remain blocked by one.
+- **4xx** — "unavailable". No rules were published, and the fetch proceeds.
+- **5xx** — "unreachable". A complete disallow, recorded as `robots.txt could
+  not be served`. This is *stricter* than before, where any non-200 that was
+  not 401/403 fell through to `allow_all`.
+
+Nothing else moves. HTTPS only, public hosts re-checked at every redirect, two
+seconds between requests to one host, the 15 MB cap, and the PDF check all
+stand. **A 401 or 403 on the document itself is still `blocked`, and is still
+never worked around** — which is what keeps UnitedHealthcare's 229 documents
+and Mercy's 6 out, on their own servers' say-so rather than on an inference
+from a different file.
+
+### Why this is not a workaround
+
+ADR 0013's rule stands on "as robots.txt conventions go", and the convention
+it names is not the one the standard states. Reading the standard correctly is
+not getting past a refusal; it is noticing that no refusal was made. The
+project's rule — a host that says no is left alone — is unchanged, and the two
+hosts that say no are still left alone.
+
+The documents in question are Summaries of Benefits and Coverage, which
+issuers are federally required to publish for exactly this audience. That is
+context, not a licence: the deciding facts are the standard and the servers'
+own answers.
+
+### A network error still fails open, and is now logged
+
+`requests` raising means we never learned anything. RFC 9309 would group that
+with "unreachable" and disallow. It is left as it was, because changing it
+would newly block hosts on a transient blip and that effect has not been
+measured — but it now logs, instead of failing open in silence. Recorded here
+rather than left implied.
+
+## Consequences
+
+**Measured.** The eight affected states were refreshed — 1,496 documents
+re-checked — and the estimate held:
+
+| | Before | After |
+| --- | --- | --- |
+| Plans with text | 1,940 (59.2%) | **2,274 (69.4%)** |
+| Documents `ok` | 1,224 | 1,360 |
+| Documents `partial` | 22 | **138** |
+| Documents `blocked` | 995 | 748 |
+
+The run's outcomes were `698 unchanged, 434 blocked, 136 ok, 116 partial,
+77 changed:ok, 31 http_error, 4 unreachable`.
+
+- **The documents this reaches parse less completely than the ones already
+  read.** Partial went from 22 to 138: 116 of this run's documents gave up
+  their text but not all of the template. They are honestly marked and their
+  text is searched (ADR 0017), but "69.4% covered" means 2,136 whole documents
+  and 138 partial ones, not 2,274 whole ones. Worth saying plainly rather than
+  reporting the headline alone.
+- **UnitedHealthcare and Mercy now fail on their own status.** `www.uhc.com`
+  returns 403 for the document, so those 125 records moved from "robots.txt
+  refused (HTTP 403)" to "HTTP 403" — the same `blocked` status and the same
+  user-facing sentence, now for the accurate reason. 110 records in the ten
+  states not refreshed still carry the old wording; the next monthly refresh
+  clears them, and the status they resolve to is unchanged either way.
+- **Nothing was deleted (ADR 0016), verified.** 1,264 PDFs before, 1,593
+  after; `comm` confirms every file present before is still present, and the
+  archive grew by exactly 77 — the count of documents the issuer had replaced.
+- **The reason a plan has no document gets more honest.** "The insurer blocks
+  automated access" now means the insurer's server refused the document, not
+  that it refused an unrelated file.
+- **`robots.txt refused (HTTP 401/403)` disappears as a stored reason.** A
+  refresh re-judges those documents; the ones that still refuse land on the
+  document's own status.
+- **A 5xx now blocks where it used to pass.** Rare, and the safer direction.
+- **This is re-measurable.** Seven hosts, one request each — the same census
+  can be re-run whenever the question comes up again.

src/ingestion/sbc/fetch.py

@@ -3,8 +3,9 @@
 The URLs come from CMS, not users, but they point at dozens of issuers'
 servers, so each is checked before it is fetched and again at every redirect:
 HTTPS only, and never an IP address or a local host name. A site that
-refuses us — robots.txt, or a 401/403 — is recorded as `blocked` and left
-alone. Nothing here tries to get past it.
+refuses us — a robots.txt rule, or a 401/403 on the document itself — is
+recorded as `blocked` and left alone. Nothing here tries to get past it. A
+robots.txt that cannot be served is not a refusal (ADR 0020).
 
 `revalidate` asks whether a stored document has changed, with the validators
 the issuer gave us, and writes nothing.
@@ -163,24 +164,36 @@ def _unsafe(url: str) -> str | None:
 
 
 @lru_cache(maxsize=256)
-def _robots(origin: str) -> RobotFileParser | int:
-    """The site's robots.txt rules, read once per run, or the 401/403 it answered with.
+def _robots(origin: str) -> RobotFileParser:
+    """The site's robots.txt rules, read once per run (RFC 9309 §2.3.1, ADR 0020).
 
     Read with our own client, not RobotFileParser.read(), for the timeout and
-    User-Agent. As robots.txt conventions go: 401/403 means stay out, any
-    other failure means no rules were published.
+    User-Agent. The status decides what the site told us:
+
+    - 2xx: these are the rules, and they are obeyed.
+    - 4xx "unavailable": the file is not there to read, so no rules were
+      published and the standard says a crawler may fetch. A 401 or 403 on
+      `/robots.txt` says nothing about the documents themselves.
+    - 5xx "unreachable": the server is failing, and the standard says assume a
+      complete disallow rather than guess.
+
+    A network error is the one place this still fails open, which RFC 9309
+    would group with unreachable. It is left as it was and logged rather than
+    changed unmeasured (ADR 0020).
     """
     parser = RobotFileParser()
     _pace(urlsplit(origin).hostname)
     try:
         response = requests.get(f"{origin}/robots.txt", headers={"User-Agent": USER_AGENT},
                                 timeout=_TIMEOUT, allow_redirects=True)
-    except requests.RequestException:
+    except requests.RequestException as exc:
+        logger.info("robots.txt unreachable for %s (%s): proceeding", origin, type(exc).__name__)
         parser.allow_all = True
         return parser
-    if response.status_code in (401, 403):
-        return response.status_code
-    if response.status_code != 200:
+
+    if response.status_code >= 500:
+        parser.disallow_all = True
+    elif response.status_code != 200:
         parser.allow_all = True
     else:
         parser.parse(response.text.splitlines())
@@ -191,8 +204,8 @@ def _robots_refusal(url: str) -> str | None:
     """Why the site's robots.txt keeps us from `url`, or None."""
     parts = urlsplit(url)
     rules = _robots(f"{parts.scheme}://{parts.netloc}")
-    if isinstance(rules, int):
-        return f"robots.txt refused (HTTP {rules})"
+    if rules.disallow_all:
+        return "robots.txt could not be served"
     return None if rules.can_fetch(USER_AGENT, url) else "robots.txt disallows it"
 
 

tests/test_sbc_fetch.py

@@ -99,15 +99,37 @@ def test_robots_txt_is_honoured(_offline):
     assert [call.args[0] for call in _offline.call_args_list] == ["https://sbc.example.com/robots.txt"]
 
 
-def test_a_site_refusing_robots_txt_is_left_alone(_offline):
-    _serve(_offline, _response(), robots=_response(403))
+@pytest.mark.parametrize("status", [401, 403, 404, 410])
+def test_a_robots_txt_that_is_not_there_publishes_no_rules(_offline, status):
+    """RFC 9309 §2.3.1.3: 4xx is "unavailable", and a crawler may fetch (ADR 0020).
+
+    Measured over the seven hosts this affected, five served their PDF at 200
+    with `application/pdf`. A 401 or 403 on /robots.txt is the web server
+    declining to serve one file, and says nothing about the documents.
+    """
+    _serve(_offline, _response(), robots=_response(status))
+
+    assert fetch_pdf(URL, 2026).status == "ok"
+
+
+@pytest.mark.parametrize("status", [500, 503])
+def test_a_robots_txt_the_server_cannot_serve_stops_the_fetch(_offline, status):
+    """RFC 9309 §2.3.1.4: 5xx is "unreachable", and a crawler assumes a full disallow."""
+    _serve(_offline, _response(), robots=_response(status))
 
     result = fetch_pdf(URL, 2026)
 
-    assert (result.status, result.detail) == ("blocked", "robots.txt refused (HTTP 403)")
+    assert (result.status, result.detail) == ("blocked", "robots.txt could not be served")
     assert [call.args[0] for call in _offline.call_args_list] == ["https://sbc.example.com/robots.txt"]
 
 
+def test_a_site_that_refuses_the_document_itself_is_left_alone(_offline):
+    """The refusal that still counts: a 403 on the PDF, not on /robots.txt."""
+    _serve(_offline, _response(403))
+
+    assert fetch_pdf(URL, 2026).status == "blocked"
+
+
 def test_a_body_past_the_size_cap_is_dropped_unsaved(_offline):
     _serve(_offline, _response(body=b"%PDF-" + b"x" * MAX_BYTES))