Add CCO scraper

[NisargOza]

What this does

Adds a Cancer Care Ontario (CCO) scraper to the datasets scraping pipeline. Same shape as the NICE scraper — discovery and extraction are separate, and everything outputs a normalized ScrapedDocument.

Unlike NICE, CCO sits behind an Azure WAF JS challenge that rejects plain HTTP requests regardless of headers, so this scraper drives a real headless browser (Playwright) instead of httpx. Every function that touches the network takes a plain fetch(url) -> html callable rather than a browser handle directly, so discovery and parsing stay unit-testable without a browser.

The scraper extracts each guideline's short summary sections (objective, patient population, intended users, etc.) and bibliographic metadata (ID, version, type of content, document status, authors). It does not fetch the linked PDF, matching this repo's existing HTML-over-PDF stance for NICE — the PDF is recorded as a markdown link and as metadata["pdf_url"] for a later PDF-aware pass. It also handles several distinct page templates spanning roughly two decades of CCO's CMS history, including guidelines with no narrative sections at all, and guidelines archived 5+ years where CCO itself hides the abstract and PDF behind an HTML comment.

scrape_listing_documents in base.py is generalized to accept any client type, not just httpx.Client (NICE is unaffected — its type is inferred the same as before).

How discovery works

Discovery starts from CCO's types-of-cancer index page, which is itself the full master listing of every guideline — the per-cancer-type category buttons on it are just filtered subsets of the same list, so they aren't crawled separately. The listing is paged through a Drupal "Load more" pager rather than a rel=next link, so pagination follows that pager's markup directly.

Discovery retries itself up to 3 times if it comes back with a suspiciously small number of guidelines (fewer than 50, when CCO reports several hundred) — a thin first result turned out to be a load-timing issue rather than a real "no guidelines" state. If it's still thin after retries, it raises loudly rather than silently reporting a false "0 documents scraped" success.

How extraction works

Each guideline page is fetched through Playwright (resolving CCO's WAF challenge the same way a real browser would) and parsed with lxml. Bibliographic fields (Version, ID, Type of Content, Document Status, Authors) and the PDF link are matched against the whole page — they're precise, unambiguous matches, and at least one CCO template puts this metadata block outside the page's main-content region entirely. Narrative sections (Guideline Objective, Patient Population, Intended Guideline Users, Research Question(s), etc.) are matched preferentially within the page's role="main"/#content region, falling back to the whole page only if that comes up empty — this stops global nav/footer boilerplate from being absorbed into whichever section happens to be last on the page.

The extraction preserves:

• bibliographic metadata (version, ID, type of content, document status, authors), both as structured metadata and as inline content when no narrative sections exist
• narrative sections (objective, patient population, intended users, research questions, etc.)
• the linked full-report PDF as a markdown link and as metadata["pdf_url"]
• the page title, falling back to the page's <h1> when CCO's own <title> tag comes back empty (observed on some archived pages)

Guidelines that fail to parse (an unrecognized template, a fetch error, a persistent WAF-challenge failure) don't abort the run — they're yielded as a placeholder document with metadata["scrape_error"] set, so a run across ~400 guidelines spanning many CMS eras can complete even if a handful of pages don't match anything recognized.

CLI

uv run playwright install chromium # one-time browser binary download
uv run amfv-scrape --source cco --documents 1
uv run amfv-scrape --source cco --url https://www.cancercareontario.ca/en/guidelines-advice/types-of-cancer/64736
uv run amfv-scrape --source cco --documents 5 -f markdown -o ./cco-out/
uv run amfv-scrape --source cco --documents all -o cco.jsonl

--source all now includes CCO alongside NICE.

Licensing

CCO / Ontario Health guideline content appears on Meditron's list of redistributable sources for this kind of pipeline, but we haven't independently confirmed a redistribution license directly from CCO/Ontario Health. Treat bulk reuse as needing the same confirmation step as any other source until that's verified.

QA

• CCO's own listing page reports ~400 guidelines under the "Guidelines & Advice" content type; a full run discovered 379 unique detail pages, with real (non-placeholder) content for the large majority. The remaining gap is mostly accounted for by archived-guideline pages where CCO itself has hidden the body content, plus a handful of flagged placeholders visible via metadata["scrape_error"].
• Confirmed at least three distinct page templates in production: the standard "Advice" template (inline objective/population/users sections), a thin PEBC-style template (bibliographic metadata only, no narrative sections), and an archived (5+ years) template where the abstract and PDF link are HTML-commented-out and the metadata block sits outside the main-content region.

Test plan

• uv run pytest datasets/test/test_scraping_cco.py — URL validation, field/section parsing across all three known templates, Load-more pagination, WAF-challenge and transient-error retries, discovery sanity-check retry, per-item failure resilience
• uv run pytest — no regressions (39 passed)
• uv run ruff check . / uv run ruff format --check .
• Reviewer runs markdown output: uv run amfv-scrape --source cco --documents 5 -f markdown -o /tmp/cco-out/
• Reviewer runs a fresh full scrape (--documents all) to confirm discovered/scraped counts and spot-check any placeholder records

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}