fix: route reference URLs through source_kind, not just github.com - #133
Conversation
CodeInstructPipeline._build_task and EquivalenceTestsPipeline._build_task both built the "reference" provenance URL as a hardcoded github.com blob link. Neither pipeline sets required_capabilities, so both run against GitHub, GitLab, and local sources per sources.py's capability table, and a GitLab-sourced repo got a 404 while a local checkout got a link to a host it was never on. Add sources.blob_reference_url, following the same source_kind branch huggingface#99 added to commit_runtime: github.com/blob with #L10-L20 anchors, gitlab.com/-/blob with #L10-20 anchors (GitLab's canonical line-range form has no second L), and None for a local checkout, since the field is omitted rather than nulled (TOML has no null). Closes huggingface#113
KNambiarDJsc
left a comment
There was a problem hiding this comment.
I filed #113, this closes it — thanks for picking it up.
Reviewed and tested 85ab1ef:
- The GitLab URL format is correct. I checked it against a real repo:
gitlab.com/<ns>/<project>/-/blob/<ref>/<path>#L{start}-{end}(canonical/-/blob/form, singleLin the anchor) resolves with a real HTTP 200 — matches what this PR emits exactly. - Local-checkout
None-omission is handled correctly, same pattern as #99: I ran the dedicatedwrite_harbor_taskroundtrip test myself and confirmed the emittedtask.tomlnever contains areference = "null"(whichtomli_wwould otherwise reject outright). - Test run:
code_instruct+equivalence_tests+sources+commit_runtimesuites: 99 passed. Broader related coverage (all pipeline tests + hub metadata): 203 passed.ruff check/ruff format --check: clean.
LGTM. One non-blocking note, same limitation I flagged on #99 and inherited here since both reuse RepoSpec.owner_name: nested GitLab groups (gitlab.com/group/subgroup/project) only keep the last two path segments, so owner_name returns ("subgroup", "project") and the emitted link drops the outer group — 404s for a repo under a nested namespace. Not this PR's scope to fix (same root cause, same call site as #99), just flagging the pattern is now in three places (commit_runtime, code_instruct, equivalence_tests) in case a shared fix is ever worth doing.
|
Thanks for testing the GitLab URL against a real repo, that's exactly the check I couldn't do myself. Agreed on the nested-group limitation, same root cause as #99 and out of scope here. Worth a shared fix in |
|
Thanks, tested and merged. This fixes the GitLab/local reference links. We'll track nested GitLab groups separately. |
|
Glad it's in. Nested groups makes sense to split out, same RepoSpec.owner_name root cause as #99, so it'll want the same fix rather than a special case here. |
Summary
CodeInstructPipeline._build_taskandEquivalenceTestsPipeline._build_taskboth hardcoded agithub.comblob URL for the task'sreferencefield. Neither pipeline setsrequired_capabilities, so both run against any source (GitHub, GitLab, or a local checkout), and a GitLab-sourced repo got a link that 404s; a local checkout got a link to a host it was never on.sources.blob_reference_url, which branches onsource_kindthe same way fix(commit_runtime): use the real host in the dataset reference URL #99 did forcommit_runtime:github.com/.../blob/...#L{a}-L{b}, GitLab's canonicalgitlab.com/.../-/blob/...#L{a}-{b}(no secondL), andNonefor a local checkout so the field is omitted rather than written asnull(TOML has no null).Test plan
pytest tests/test_pipeline_code_instruct.py tests/test_pipeline_equivalence_tests.py tests/test_pipeline_commit_runtime.py tests/test_sources.pyon Python 3.12 and 3.14, in Docker: 97 passed, 2 skipped (the 2 skips are pre-existing, gated ongitnot being installed in the image).github.com, and raise on the local case) and pass after the fix; verified both ways in the same containers.ruff checkandruff format --checkon the changed files: clean.Out of scope
cve_patches.pyhas the same-looking hardcodedgithub.comstring, but it declaresrequired_capabilities = {Capability.COMMIT_API}, which onlySourceKind.GITHUBhas, so it can never run against GitLab or local. Left alone.Closes #113