Question about auto_map usage in config.json for LlamaEncoderModel

[ly1998117]

Hi, thanks for your great work on LLM2CLIP

I noticed that in the config.json of microsoft/LLM2CLIP-Llama-3-8B-Instruct-CC-Finetuned, there is an auto_map field pointing to:

"auto_map": {
"AutoModel": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp--modeling_llama_encoder.LlamaEncoderModel"
}

This makes AutoModel.from_pretrained() load the model class from the McGill-NLP repository via trust_remote_code=True and HuggingFace’s dynamic module mechanism.

However, this repo already includes a local implementation of LlamaEncoderModel in modeling_llama_encoder.py. I tried deleting auto_map and loading the checkpoint with the local class instead. The model loads successfully with no missing or unexpected keys.

May I ask what the original motivation was for adding this cross-repo auto_map? Is there any functional reason the model must be loaded from
the McGill-NLP repo rather than using the local implementation here?

Thanks again for the great work!

[raytrun]

Thanks for pointing this out.

There’s no functional reason to load the model from the McGill-NLP repository. The cross-repo auto_map was added early on mainly for convenience, to avoid copying the encoder code at the time.

The local modeling_llama_encoder.py in this repo works perfectly with the released checkpoints, and the model can now be loaded using the local implementation.

So it makes sense to update the config to use the local implementation and remove the external dependency.