Skip to content

fix: route reference URLs through source_kind, not just github.com - #133

Merged
adithya-s-k merged 1 commit into
huggingface:mainfrom
AmirF194:fix/113-reference-url-source-kind
Sep 17, 2026
Merged

adithya-s-k merged 1 commit into
huggingface:mainfrom
AmirF194:fix/113-reference-url-source-kind

Conversation

@AmirF194

Copy link
Copy Markdown
Contributor

Summary

  • CodeInstructPipeline._build_task and EquivalenceTestsPipeline._build_task both hardcoded a github.com blob URL for the task's reference field. Neither pipeline sets required_capabilities, so both run against any source (GitHub, GitLab, or a local checkout), and a GitLab-sourced repo got a link that 404s; a local checkout got a link to a host it was never on.
  • Added sources.blob_reference_url, which branches on source_kind the same way fix(commit_runtime): use the real host in the dataset reference URL #99 did for commit_runtime: github.com/.../blob/...#L{a}-L{b}, GitLab's canonical gitlab.com/.../-/blob/...#L{a}-{b} (no second L), and None for a local checkout so the field is omitted rather than written as null (TOML has no null).

Test plan

  • pytest tests/test_pipeline_code_instruct.py tests/test_pipeline_equivalence_tests.py tests/test_pipeline_commit_runtime.py tests/test_sources.py on Python 3.12 and 3.14, in Docker: 97 passed, 2 skipped (the 2 skips are pre-existing, gated on git not being installed in the image).
  • The two new parametrized tests fail on the pre-fix code (still hardcode github.com, and raise on the local case) and pass after the fix; verified both ways in the same containers.
  • ruff check and ruff format --check on the changed files: clean.

Out of scope

  • cve_patches.py has the same-looking hardcoded github.com string, but it declares required_capabilities = {Capability.COMMIT_API}, which only SourceKind.GITHUB has, so it can never run against GitLab or local. Left alone.

Closes #113

CodeInstructPipeline._build_task and EquivalenceTestsPipeline._build_task
both built the "reference" provenance URL as a hardcoded github.com blob
link. Neither pipeline sets required_capabilities, so both run against
GitHub, GitLab, and local sources per sources.py's capability table, and
a GitLab-sourced repo got a 404 while a local checkout got a link to a
host it was never on.

Add sources.blob_reference_url, following the same source_kind branch
huggingface#99 added to commit_runtime: github.com/blob with #L10-L20 anchors,
gitlab.com/-/blob with #L10-20 anchors (GitLab's canonical line-range
form has no second L), and None for a local checkout, since the field
is omitted rather than nulled (TOML has no null).

Closes huggingface#113

@KNambiarDJsc KNambiarDJsc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #113, this closes it — thanks for picking it up.

Reviewed and tested 85ab1ef:

  • The GitLab URL format is correct. I checked it against a real repo: gitlab.com/<ns>/<project>/-/blob/<ref>/<path>#L{start}-{end} (canonical /-/blob/ form, single L in the anchor) resolves with a real HTTP 200 — matches what this PR emits exactly.
  • Local-checkout None-omission is handled correctly, same pattern as #99: I ran the dedicated write_harbor_task roundtrip test myself and confirmed the emitted task.toml never contains a reference = "null" (which tomli_w would otherwise reject outright).
  • Test run: code_instruct + equivalence_tests + sources + commit_runtime suites: 99 passed. Broader related coverage (all pipeline tests + hub metadata): 203 passed. ruff check / ruff format --check: clean.

LGTM. One non-blocking note, same limitation I flagged on #99 and inherited here since both reuse RepoSpec.owner_name: nested GitLab groups (gitlab.com/group/subgroup/project) only keep the last two path segments, so owner_name returns ("subgroup", "project") and the emitted link drops the outer group — 404s for a repo under a nested namespace. Not this PR's scope to fix (same root cause, same call site as #99), just flagging the pattern is now in three places (commit_runtime, code_instruct, equivalence_tests) in case a shared fix is ever worth doing.

@AmirF194

Copy link
Copy Markdown
Contributor Author

Thanks for testing the GitLab URL against a real repo, that's exactly the check I couldn't do myself.

Agreed on the nested-group limitation, same root cause as #99 and out of scope here. Worth a shared fix in RepoSpec.owner_name once it's hit a third time; not doing it in this PR.

@adithya-s-k
adithya-s-k merged commit 9cbc429 into huggingface:main Sep 17, 2026
11 checks passed
@adithya-s-k

Copy link
Copy Markdown
Collaborator

Thanks, tested and merged. This fixes the GitLab/local reference links. We'll track nested GitLab groups separately.

@AmirF194

Copy link
Copy Markdown
Contributor Author

Glad it's in. Nested groups makes sense to split out, same RepoSpec.owner_name root cause as #99, so it'll want the same fix rather than a special case here.

@AmirF194
AmirF194 deleted the fix/113-reference-url-source-kind branch September 17, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

code_instruct and equivalence_tests also hardcode github.com in the reference URL for GitLab-sourced repos

3 participants