Repository navigation
Kernels subpackage and huggingface_hub patcher migration - #79
Merged
tastelikefeet merged 6 commits intoNov 3, 2026
Merged
Conversation
tastelikefeet
commented
Nov 2, 2026
tastelikefeet
commented
Collaborator
- Add modelscope_hub.kernels: a drop-in wrapper around the kernels library that loads optimized compute kernels from ModelScope (factory swap on kernels.hf_hub._get_hf_api, v1 fallback for branch-less mirrors, trusted-publisher allowlist via MS_KERNELS_TRUSTED_PUBLISHERS).
- Add modelscope_hub.compat.hf_hub_patcher: monkey-patches the huggingface_hub module functions (hf_hub_download, file_exists, whoami, create_repo, upload_file, upload_folder, create_commit, repocard) onto ModelScope-backed implementations, with the legacy compatibility contract kept stable (whoami dict, RepoUrl-shaped str, commit_url shape, master default revision, Add-only create_commit) and idempotent markers shared with the legacy modelscope patcher.
- Uploads go through the commit API (resumable, LFS-routed) instead of a git clone + push; non-model repo_type now fails fast instead of silently uploading to a model repository.
- New extras: kernels (kernels>=0.17), hf (huggingface-hub>=0.19, verified against 0.19.4 source audit and 1.32 test suite).
- Export patch helpers from modelscope_hub.compat; document both features in the README.
- Add modelscope_hub.kernels: a drop-in wrapper around the kernels library that loads optimized compute kernels from ModelScope (factory swap on kernels.hf_hub._get_hf_api, v1 fallback for branch-less mirrors, trusted-publisher allowlist via MS_KERNELS_TRUSTED_PUBLISHERS). - Add modelscope_hub.compat.hf_hub_patcher: monkey-patches the huggingface_hub module functions (hf_hub_download, file_exists, whoami, create_repo, upload_file, upload_folder, create_commit, repocard) onto ModelScope-backed implementations, with the legacy compatibility contract kept stable (whoami dict, RepoUrl-shaped str, commit_url shape, master default revision, Add-only create_commit) and idempotent markers shared with the legacy modelscope patcher. - Uploads go through the commit API (resumable, LFS-routed) instead of a git clone + push; non-model repo_type now fails fast instead of silently uploading to a model repository. - New extras: kernels (kernels>=0.17), hf (huggingface-hub>=0.19, verified against 0.19.4 source audit and 1.32 test suite). - Export patch helpers from modelscope_hub.compat; document both features in the README.
- New scoped entry points in modelscope_hub.kernels: install_kernel (cache-only download for CI pre-install / offline preparation), install_kernel_all_variants (every build variant plus the dependency tree; resolved from the kernels.install submodule since it is not exported on the kernels top level), get_locked_kernel (lockfile-pinned load), and load_kernel (import an already-installed locked kernel). - Kernel lockfiles record hub-specific commit hashes: a lockfile created against the Hugging Face Hub cannot resolve on ModelScope. This is a semantic constraint, not a defect, so it is documented in the wrapper docstrings and the README, and the shim now attaches a PEP 678 note (Python 3.11+, silently skipped otherwise) when a full commit-hash revision cannot be found, explaining how to re-lock in the ModelScope environment. Exception types are unchanged, so kernels' own except clauses keep working.
Development version for the upcoming 0.7.0 release (kernels subpackage and huggingface_hub patcher). A dev build on PyPI lets downstream requirements pin >=0.7.0.dev0 resolve before the final 0.7.0 tag.
kernels 0.18 validates ref target_commits with Oid.from_str (40/64 hex digits) and both kernels 0.17 and 0.18 use target_commit as the download revision, so filling in the branch name made version resolution fail on kernels 0.18 with "Invalid git object id ...: master". ModelScope's revision API does not expose git commit hashes, so list_repo_refs now hands out a deterministic synthetic OID per (repo, ref) and every download/list entry point (snapshot_download, hf_hub_download, list_repo_tree, file_exists) maps a known synthetic OID back to the real ModelScope branch/tag name before forwarding. Real branch/tag revisions pass through untouched, and an unknown hex (e.g. from a cross-hub lockfile or a stale cached ref) still fails with the existing diagnostic note. Verified against kernels 0.17.2 and 0.18.1 end to end on the kernels-community mirrors (42 variants resolved via the master->v1 fallback on both).
wangxingjun778
approved these changes
Nov 3, 2026
modelscope-hub no longer declares any relationship to huggingface_hub in its metadata: the `hf` extra is removed. The patcher entry points (patch_hf_hub / hf_hub_patch_scope) now check for huggingface_hub when used and raise an actionable ImportError with installation instructions when it is missing; unpatch_hf_hub stays a silent no-op in minimal environments (nothing can have been patched). kernels remains untouched: its own dependency chain pulls huggingface_hub, and the kernels shim only runs inside that library. huggingface_hub>=0.19 remains the documented minimum in the module docstring and the README.
Per the product decision, the huggingface_hub monkey-patcher is not migrated to modelscope_hub: compat/hf_hub_patcher.py, its exports and tests are removed, and the README section is dropped. modelscope_hub 0.7.0 now ships exactly one integration feature, the kernels package (modelscope_hub.kernels), which the modelscope package's patch_hub() keeps delegating to. The synthetic-OID ref handling, variant discovery, convenience wrappers and the kernels tests are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.