[critical] fix: [html_to_markdown] close SSRF holes in the URL safety check

[elhoim]

BLUF — fetchHTML follows redirects unchecked and misses four blocked ranges, allowing SSRF.

• Problem — fetchHTML in html_to_markdown.py validated a URL once and then let requests follow redirects automatically, so an allowed external URL replying with a 302 to an internal address (Redis, metadata endpoints, admin panels) was fetched transparently, and the hand-rolled blocked-range list missed fe80::/10, fc00::/7, 0.0.0.0/8 and 100.64.0.0/10.
• Fix — Disables automatic redirects, loops over hops itself with urljoin and re-runs is_safe_url on each of at most five hops, and replaces the range list with the ipaddress module's built-in address classification.
• Effect — An analyst enriching an attacker-supplied URL can no longer make the MISP server reach internal-only services.

The defect

fetchHTML validated the URL once, then let requests follow redirects automatically:

def fetchHTML(url):
    if not is_safe_url(url):
        raise ValueError(f"Blocked URL: {url}")
    r = requests.get(url, timeout=10)
    return r.text

A request to an allowed external URL that responds with a 302 to http://127.0.0.1:6379/ (or any other internal address) is followed transparently — the safety check only ever saw the original, safe URL. Separately, the IP-blocking helper used a hand-rolled list:

BLOCKED_RANGES = [
    ipaddress.ip_network("127.0.0.0/8"),
    ipaddress.ip_network("10.0.0.0/8"),
    ipaddress.ip_network("172.16.0.0/12"),
    ipaddress.ip_network("192.168.0.0/16"),
    ipaddress.ip_network("169.254.0.0/16"),
    ipaddress.ip_network("::1/128"),
]

which omits fe80::/10, fc00::/7, 0.0.0.0/8, and 100.64.0.0/10 (carrier-grade NAT), so URLs like http://[fe80::1]/ or http://0.0.0.0/ passed the check outright.

Impact

The html_to_markdown module runs on MISP server infrastructure. An analyst who enriches an attribute containing an attacker-controlled URL can be made to have the MISP server issue requests to internal-only services (metadata endpoints, Redis, internal admin panels, link-local services) via either an open redirect or one of the unblocked address ranges, and have the response content returned into MISP as "markdown" — a classic internal SSRF against the MISP host itself.

The fix

fetchHTML now disables automatic redirect following (allow_redirects=False) and loops over redirect hops itself, re-resolving relative Location headers with urljoin and re-running is_safe_url on every hop, capped at 5 redirects. _is_ip_blocked now uses not ipaddress.ip_address(...).is_global instead of the hand-rolled range list, which additionally blocks 0.0.0.0/8, 100.64.0.0/10, fe80::/10, and fc00::/7.

Note: the finding's suggested replacement, is_private or is_loopback or is_link_local or is_reserved, does not actually block 100.64.0.0/10 on CPython (verified in the project venv — that range is is_private but not flagged by the others either), so is_global was used instead since it correctly covers the full non-global address space in one check.

Verification

• flake8 --extend-exclude=misp_modules/lib/,tests/,website/ on the changed file: clean, no output.
• Full module test suite: 161 passed, 4 skipped, 5 subtests passed in 58.69s.

Found during a review of the repository; other findings are being submitted as separate PRs.

🤖 Generated with Claude Code

[elhoim]

[pr-sweep] superseded by #908 - the base changed substantially, so this has been rebuilt cleanly on current main.

Upstream f359a02 and 2448f74 already fixed the redirect half of this PR. #908 carries only the part that is still open: the missing 0.0.0.0/8, 100.64.0.0/10, ::/128, fc00::/7 and fe80::/10 entries in BLOCKED_RANGES.