mmcontext

Multimodal contrastive learning aligning text and omics embeddings via sentence-transformers.

mmcontext enables joint embedding spaces where natural-language descriptions and biological data (single-cell gene expression) can be compared directly. Built on the sentence-transformers v5.4+ multimodal API.

Paper

mmcontext is described in detail in our paper: Jonatan Menger, Sonia Maria Krissmer, Clemens Kreutz, Harald Binder, Maren Hackenberg, mmContext: an open framework for multimodal contrastive learning of omics and text data, Bioinformatics, Volume 42, Issue 6, June 2026, btag338, https://doi.org/10.1093/bioinformatics/btag338 If you use mmcontext in your research, please cite our work.

Installation

pip install mmcontext

Or install from source with development dependencies:

git clone https://github.com/mengerj/mmcontext.git
cd mmcontext
pip install -e ".[dev,test,eval,train]"

Quick Start

Using a Pre-trained Model

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("jo-mengr/mmcontext-pubmedbert-gs10k", trust_remote_code=True)

# Encode text queries
text_embeddings = model.encode(["CD4+ T cell", "B cell precursor"])

Pre-trained models are available on Hugging Face under the jo-mengr organization.

See the pretrained inference tutorial for a complete guide.

Building a Pipeline

from mmcontext.embed import build_pipeline

pipeline = build_pipeline(
    text_model="microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract",
    omics_dim=512,        # dimension of your omics vectors
    shared_dim=256,       # joint embedding space dimension
)

Evaluation

from mmcontext.eval import get

LabelSimilarity = get("LabelSimilarity")
evaluator = LabelSimilarity(similarity="cosine")
result = evaluator.compute(
    omics_embeddings=omics_emb,
    label_embeddings=label_emb,
    query_labels=labels,
    true_labels=true_labels,
    label_key="cell_type",
)

See the evaluate model 2.0 notebook for a complete evaluation workflow.

Architecture

mmcontext 2.0 uses the sentence-transformers module pipeline pattern:

Input -> MMContextModule -> AdapterModule -> Pooling -> Normalize

The MMContextModule handles both text (tokenize -> AutoModel) and omics (VectorStore lookup) modalities. The AdapterModule projects omics vectors into the text model's embedding space.

Data is stored in AnnData format (.h5ad/zarr), with a memory-efficient VectorStore for runtime omics lookups.

Multi-Model Benchmarking

For systematic comparisons across multiple models and datasets, see the companion repository: mmcontext-benchmark.

Dataset Preparation

To prepare training datasets from AnnData objects, see adata-hf-datasets.

Contributing

Contributions and suggestions are very welcome! Please open an issue to propose enhancements, report bugs, or ask questions.

Citation

@article{menger2026mmcontext,
  title={mmContext: an open framework for multimodal contrastive learning of omics and text data},
  author={Menger, Jonatan and Krissmer, Sonia Maria and Kreutz, Clemens and Binder, Harald and Hackenberg, Maren},
  journal={Bioinformatics},
  volume={42},
  number={6},
  pages={btag338},
  year={2026},
  publisher={Oxford University Press}
}

---