[Badri-Rajendran]

Problem

ADR 0013 recorded a site that would not serve /robots.txt as blocked:

A 401 or 403, on the file or on robots.txt, records the document as blocked. A refused robots.txt counts as disallowing everything.

That was written from two hosts in a two-state slice. At eighteen states it governs 482 documents — the second largest reason for missing coverage after a real Disallow rule.

It is also stricter than the standard it implements. RFC 9309 §2.3.1.3 puts every 400–499 in one class, "Unavailable", and says a crawler may access the site's resources; only §2.3.1.4's "Unreachable" (5xx) requires assuming a complete disallow. A 401 or 403 on /robots.txt is a web server declining to serve one file — not a statement about the documents.

The measurement

The cost of the rule had never been measured. All 482 documents come from seven hosts, so this is a census, not a sample: one request each, two seconds apart, status and content type only, nothing stored or parsed.

Host	Docs	/robots.txt	The PDF itself
www.uhc.com	229	403	403
portal.medica.com	117	403	200 application/pdf
sbc.anthem.com	76	401	200 application/pdf
sbc.wellpoint.com	34	401	200 application/pdf
apps.avmed.com	14	401	200 application/pdf
www.mercyhealthsystem.org	6	403	403
static.mending.com	6	403	200 application/pdf

Five of seven serve the document without objection. My prediction going in was the opposite — that a 403 on robots.txt meant a WAF refusing the client outright, so the PDF would refuse too. That holds for exactly two hosts, and both already refuse on their own account.

Implementation

_robots() now follows the status, as RFC 9309 describes:

• 2xx — the rules, parsed and obeyed. Unchanged: 483 documents stay blocked by a real Disallow.
• 4xx — "unavailable": no rules published, the fetch proceeds.
• 5xx — "unreachable": a complete disallow, recorded as robots.txt could not be served. Stricter than before, where any non-200 that was not 401/403 fell through to allow_all.

Nothing else moves. HTTPS only, public hosts re-checked at every redirect, 2 s per-host pacing, the 15 MB cap, and the PDF check all stand. A 401 or 403 on the document itself is still blocked and still never worked around — which is exactly what keeps UnitedHealthcare's 229 and Mercy's 6 out, on their own servers' say-so rather than on an inference from a different file.

A network error still fails open, as it did, but now logs instead of doing it silently. ADR 0020 records that as a deliberate, unmeasured divergence rather than changing it blind.

Validation

• uv run pytest -q — 406 passing. tests/test_sbc_fetch.py goes from 16 to 33 tests: 4xx parametrised over 401/403/404/410 proceeds, 5xx over 500/503 blocks with the new reason, and a 403 on the PDF itself still blocks.
• ruff check, bandit -r src -ll clean. No migration.
• ADR 0020 written; ADR 0013's superseded clause struck through in place, with its status block updated.

Security / DB / API impact

• No schema, endpoint or API change.
• Politeness posture: unchanged in every respect except the one the standard defines differently. One 5xx case became stricter. Refusals of the document are still absolute.
• Re-running the eight affected states now to record the real coverage change; the ADR's estimate is 247 documents behind 333 plans, taking coverage from 1,940 (59.2%) to about 2,273 (69.4%).

Note

Stacked on #31 → #30 → #29 → #28.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.