Skip to content

Support recursive repository file discovery and options for local repositories - #473

Closed
dkuku wants to merge 3 commits into
elixir-nx:mainfrom
dkuku:feature/recursive-repo-files
Closed

dkuku wants to merge 3 commits into
elixir-nx:mainfrom
dkuku:feature/recursive-repo-files

Conversation

@dkuku

@dkuku dkuku commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds support for recursive file discovery when loading repository files from HuggingFace Hub and local directories, and brings feature parity to {:local, directory, options}.

Changes

  1. HuggingFace Hub Tree Listing (HuggingFace.Hub.file_listing_url/4):

    • Added support for :recursive option, defaulting to true (passing ?recursive=true to the Hugging Face tree API).
    • Allows discovering and downloading files located within nested module or component directories (e.g. 1_Pooling/config.json, 2_Dense/model.safetensors).
    • Backwards compatible with existing 3-argument callers while allowing explicit opt-out with recursive: false.
  2. Local Directory Loading ({:local, dir} and {:local, dir, opts}):

    • get_repo_files searches up to depth 2 (dir/* and dir/*/*) when recursive: true (the default), discovering files inside module subdirectories while avoiding unbounded filesystem traversal.
    • Added support for {:local, directory, options} with :subdir and :recursive options, matching {:hf, repository_id, options}.
    • Updated download/3 and normalize_repository!/1 accordingly.
  3. Tests:

    • Added unit tests in HubTest for recursive and non-recursive file_listing_url.
    • Added unit tests in BumblebeeTest for local directory loading with nested subdirectories, :subdir, and depth-2 boundary enforcement.

@dkuku dkuku closed this Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant