add PubMed guideline scraper

[zndr27]

What this does

Adds a PubMed scraper to the datasets scraping pipeline. Same shape as the NICE scraper: discovery and extraction are separate, and everything outputs a normalized ScrapedDocument.

Guideline[pt] AND pubmed pmc open access[filter] AND English[la]

About 3,000 guidelines with full text. The other 40k Guideline records expose an abstract only, under publisher copyright.

Guideline[pt] rather than Practice Guideline[pt] because the former is a strict superset and the 352 records it adds are clinical, not administrative. English[la] drops 185, mostly French CMAJ translations of guidelines already present; every other source here is English-only anyway as a side effect of its entry URL, so this just makes it explicit.

Open question: how should the corpus handle licensing?

The PMC Open Access Subset is not uniformly Creative Commons licensed. Censused over all 2,999 records:

License	n	%	derivatives
CC BY	1,335	44.5%	unrestricted
CC0	38	1.3%	unrestricted
CC BY-NC	662	22.1%	non-commercial only
CC BY-NC-SA	62	2.1%	non-commercial only
CC BY-NC-ND	552	18.4%	restricted
publisher terms, no CC	350	11.7%	case by case

So 46% is unambiguously safe for a converted, chunked corpus, which is arguably a derivative work. In the tail, 112 records have no <license> element at all, just a line like © Springer-Verlag Tokyo 2007, and Elsevier's pandemic-era deposits grant free access under a copyright line still reading "All rights reserved". Being in the subset is not itself a grant to redistribute.

This PR records rather than decides: each document carries license, license_url, license_type, license_statement and copyright_statement, so the corpus can be filtered later without re-scraping. Happy to add a filter, though it probably wants a project-wide answer.

How it works

esearch pages by numeric offset (200 per page) and returns the total; esummary resolves each batch to PMCIDs and citation metadata in one call; efetch returns JATS XML per document. Offset paging maps straight onto list_page(client, page), and an empty page past the end terminates the run.

JATS tags are rewritten to HTML and passed through the shared html_to_markdown helper rather than writing a second serializer. Headings come from <sec> nesting depth. Reference lists and <xref> pointers are dropped as citation apparatus.

Figures and supplementary files are recorded in metadata rather than linked: JATS carries only a bare filename, and the served URL adds a CDN shard and content hash absent from the API response, so a constructed link 404s. Captions stay in the text.

Delay is 0.4s, inside NCBI's documented 3 requests/second. Calls retry with backoff on 429, which a real run does draw. Records PMC holds without a body, 0.6%, are logged and skipped.

CLI

uv run amfv-scrape --source pubmed --documents 1
uv run amfv-scrape --source pubmed --url https://pubmed.ncbi.nlm.nih.gov/42476581/
uv run amfv-scrape --source pubmed --documents 3 -f markdown -o ./pubmed-out/

--source all now includes PubMed alongside NICE. --url takes a PubMed or PMC URL; external_id always prefers the record's own PMID so both routes give the same ID.

Tests

uv run ruff check datasets
uv run ruff format --check datasets
uv run pytest datasets/test

53 tests, offline via httpx.MockTransport. Also read output for 18 guidelines sampled across the result set: median 47k characters, in the same band as a NICE guideline, but the largest is 526k, so retrieval will want chunking here.

Notes for review

• base.py is taken verbatim from add Royal Children's Hospital guideline scraper #10 so the two merge cleanly in either order. PubMed needs scrape_item -> ScrapedDocument | None for the bodyless records. Drop that commit if add Royal Children's Hospital guideline scraper #10 lands first.
• --source all is otherwise untouched: it returns inside the loop on main so only NICE runs, and add Canadian Paediatric Society statement scraper #9 and add Royal Children's Hospital guideline scraper #10 already fix that two different ways.