fix: prevent HTML-to-Markdown URL parser SSRF bypass

[adulau]

Motivation

• Close an SSRF bypass caused by a URL parser differential where urllib.parse and requests disagreed about the effective hostname for ambiguous authority strings (e.g. backslash before @).
• Ensure the SSRF validation observes the exact URL representation the HTTP client will use and prevent redirects from leading to private/loopback addresses.

Description

• Introduce _canonicalize_url() which rejects ambiguous inputs containing backslashes and returns the canonical URL as prepared by requests.Request(...).prepare().url for consistent transport semantics.
• Validate destinations using a new _is_safe_canonical_url() that parses and checks the canonicalized hostname against existing blocked ranges and resolution checks.
• Update is_safe_url() to validate the canonical representation instead of the raw input and change fetchHTML() to use the canonical URL while disabling automatic redirects and validating every redirect target before following it with a MAX_REDIRECTS limit.
• Add regression tests in tests/test_html_to_markdown.py that assert the crafted backslash authority payload is rejected before any HTTP call and that redirects to loopback addresses are blocked.

Testing

• python -m pytest tests/test_html_to_markdown.py -q — all unit tests and added regression cases passed (9 tests and 5 subtests passed).
• python -m compileall -q misp_modules/modules/expansion/html_to_markdown.py tests/test_html_to_markdown.py — compilation checks succeeded.
• New tests confirm the parser-differential payload does not result in any outgoing HTTP request and that private/loopback redirect targets are rejected.

---

Codex Task

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T12:56:48.651381Z	f359a02	PR opened
🔒 Security Review	✅ Completed 2026-09-25T12:59:44.411372Z	f359a02	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f359a02513

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve cookies while following redirects

Reuse one requests.Session for the redirect loop. Each call to the top-level requests.get creates and closes a fresh session, so cookies set by a redirect response are discarded before the next request; URLs whose 3xx response sets a session, consent, or authentication cookie required by the destination will therefore return the wrong page or fail, whereas the previous automatic redirect handling retained those cookies.

Useful? React with 👍 / 👎.