fix(input): download the raw file for GitHub and GitLab /blob/ URLs

[kevin9327]

Problem

The URL you get when you open a file on GitHub is its /blob/ page. Scanning a skill through that URL analyses GitHub's HTML file viewer instead of the file. NVIDIA's own skill-inspector skill, scanned three ways (live, --no-llm --format json, main at 4148ab3):

https://raw.githubusercontent.com/NVIDIA/SkillSpector/main/skills/skill-inspector/SKILL.md
  exit 0  components [('SKILL.md', 7213)]    issues []                            score 0   SAFE
https://github.com/NVIDIA/SkillSpector/raw/main/skills/skill-inspector/SKILL.md
  exit 0  components [('SKILL.md', 7213)]    issues []                            score 0   SAFE
https://github.com/NVIDIA/SkillSpector/blob/main/skills/skill-inspector/SKILL.md
  exit 1  components [('SKILL.md', 286718)]  issues ['AE1', 'AS3', 'E1', 'YR4']   score 67  DO_NOT_INSTALL

The last row saves the 286 KB page under the name SKILL.md, and the findings are about GitHub's page markup. GitLab behaves the same way: https://gitlab.com/gitlab-org/gitlab/-/blob/master/README.md downloads a 49,355-byte page and reports P9, while its /-/raw/ URL gives the 5,869-byte file and no findings.

Cause

InputHandler._is_git_url already sends a forge URL containing /blob/ to the file-download path rather than git clone. _download_file then fetches that URL as-is. On github.com and gitlab.com a /blob/ URL answers 200 text/html with the viewer page, so nothing downstream can tell it apart from a real file.

Fix

_raw_file_url rewrites the two forge file-page shapes before either download branch runs:

• https://github.com/<owner>/<repo>/blob/<ref>/<path> -> https://raw.githubusercontent.com/<owner>/<repo>/<ref>/<path>
• https://gitlab.com/<namespace>/<project>/-/blob/<ref>/<path> -> .../-/raw/<ref>/<path>

Both targets are already in ALLOWED_DOWNLOAD_HOSTS and still go through _validate_url_host. Neither needs a redirect, so the direct InputHandler() path (which does not follow redirects) works as well as the budgeted graph path. Every other URL, including /raw/, /tree/ and repository URLs, is returned unchanged.

Tests

tests/unit/test_input_handler.py::test_file_page_url_downloads_the_raw_file covers GitHub and GitLab, each through a plain InputHandler() and one carrying a WorkflowResourceBudget (the path resolve_input uses). It uses an httpx.MockTransport that serves the skill only at the raw URL and an HTML page for anything else, and asserts the single request went to the raw URL and the saved SKILL.md has the skill's bytes.

Before the fix (Windows 11, Python 3.12.10):

E   At index 0 diff: 'https://github.com/org/repo/blob/main/skills/demo/SKILL.md' != 'https://raw.githubusercontent.com/org/repo/main/skills/demo/SKILL.md'
E   At index 0 diff: 'https://gitlab.com/group/repo/-/blob/main/skills/demo/SKILL.md' != 'https://gitlab.com/group/repo/-/raw/main/skills/demo/SKILL.md'
FAILED tests/unit/test_input_handler.py::test_file_page_url_downloads_the_raw_file[github-direct]
FAILED tests/unit/test_input_handler.py::test_file_page_url_downloads_the_raw_file[github-workflow-budget]
FAILED tests/unit/test_input_handler.py::test_file_page_url_downloads_the_raw_file[gitlab-direct]
FAILED tests/unit/test_input_handler.py::test_file_page_url_downloads_the_raw_file[gitlab-workflow-budget]
4 failed

After: 4 passed. Every added line runs in test_input_handler.py, test_input_handler_ssrf.py and test_input_handler_bounds.py (checked with branch coverage). The pass-through return url is hit by the existing raw.githubusercontent.com download tests.

The live scans above, re-run on this branch:

https://github.com/NVIDIA/SkillSpector/blob/main/skills/skill-inspector/SKILL.md
  exit 0  components [('SKILL.md', 7213)]   issues []  score 0  SAFE
https://gitlab.com/gitlab-org/gitlab/-/blob/master/README.md
  exit 0  components [('README.md', 5869)]  issues []  score 0  SAFE

Suite

pytest -m "not integration and not provider" -p no:randomly, same machine:

	main	this branch
tests/unit, tests/opencode, tests/test_*.py	10 failed, 1721 passed	10 failed, 1725 passed
tests/nodes	3 failed, 3588 passed, 4 errors	3 failed, 3588 passed, 4 errors

The +4 are the new cases. The failures and errors are identical on both sides and Windows-only (release and compare_scan_accuracy harnesses, CRLF fixtures, a backslash inside a POSIX file name, and parametrized test IDs over the Windows environment-variable length limit).

ruff check src/ tests/: All checks passed. ruff format --check src/ tests/: 246 files already formatted.

This touches _download_file only, not resolve(), so it does not overlap with the /tree/ handling proposed in #561.

🤖 Generated with Claude Code

[rng1995]

[SkillSpector Review]

Approved at 875e6ccdcc3467806b947eb0a8b9c8c77e32a791.

I reviewed the complete change and its surrounding input-resolution, SSRF-validation, redirect, filename, and transitive-budget paths. The rewrite is narrowly scoped to the canonical GitHub and GitLab /blob/ file-page shapes, keeps both resulting hosts inside the existing download allowlist, and preserves the existing byte, time, redirect, and artifact limits. The parameterized tests cover both forges through the direct and workflow-budget paths and verify that the saved artifact contains the raw file rather than viewer HTML. The branch is based on current main, and all reported CI checks pass.