ProQuest: Protein Querying Using Semantic Technology for developing the UniProtKB LLM Query Interface
Efficiently accessing and analyzing comprehensive biological datasets remains challenging due to traditional querying complexities. To address this, we developed an intuitive, scalable query interface using advanced large language models. Our system enables users, regardless of computational expertise, to formulate natural-language queries that automatically translate into precise Solr database searches, significantly simplifying interaction with UniProtKB. Additionally, we implemented a semantic vector search for rapid protein similarity analyses, using protein embeddings generated by ProtT5 protein language model within an optimized approximate nearest-neighbor search framework (Annoy). This method significantly outperforms conventional BLAST searches, offering a speed increase of up to tenfold on GPU hardware. Functional insights are further enriched through integrated Gene Ontology analyses, providing biologically meaningful context to similarity searches. Currently, we are expanding the system using Retrieval-Augmented Generation, integrating real-time annotations from UniProt flat files to enhance contextual relevance and accuracy of generated responses. Evaluations using diverse biological queries demonstrated the robustness of our interface, highlighting its ability to mitigate intrinsic variability in LLM outputs through controlled prompt engineering and query retry mechanisms. Overall, our novel project substantially streamlines the retrieval process, facilitating quicker, more accurate exploration of protein functions, evolutionary relationships, and annotations.
Comprehensive platform for protein search, retrieval, and analysis using semantic, vector, and sequence-based technologies. Includes a FastAPI backend, React/Streamlit frontend, and extensive monitoring and analysis tools.
- Project Structure
- Prerequisites
- Installation & Setup
- Running the Application
- API Keys
- Monitoring & Analysis Tools
- Testing
- Assets & Data
- Directory Reference
- backend/: FastAPI server, core logic, models, and data processing scripts
- frontend/: React app (with Streamlit integration) for user interface
- config/: Setup, database, and environment scripts
- test/: Test scripts, evaluation, and analysis notebooks
- memory_monitoring/: Resource and performance monitoring
- backend/asset/: All major data files, embeddings, vector DBs, and reference files
- Python 3.10+
- Node.js (for frontend)
- pip (Python package manager)
- (Recommended) Virtual environment (venv, conda, etc.)
- Permissions to install packages and run scripts
cd backend
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r ../config/requirements.txt # Or install manually as neededpython ../config/setUpDatabase.py
python ../config/implementVectorDatabase.py
# (Optional) python ../config/implementVectorDatabaseFromFlatFiles.pycd ../frontend
npm installcd backend
uvicorn main:app --reload --port 8000Endpoints:
- POST /llm_query
- POST /vector_search
cd ../frontend
npm startpython -m streamlit run main.py # (if using Streamlit UI)To use all LLM features, obtain free API keys from:
- Google Generative AI: Get API key
- NVIDIA Models: Get API key
- Mistral AI: Get API key
- memory_monitoring/: Scripts and Excel files for memory usage
- test/: Extensive test and analysis scripts (e.g.,
testRAG.py,evaluateRAGResult.py,visualizePerformanceBarChart.py)
Test scripts are in the test/ directory. Example:
cd test
python testRAG.pyAll major data files are in backend/asset/:
- protein_index2.db: Main SQLite database
- protein_embeddings_2.ann: Annoy index for vector search
- chroma_uniprot_nomic/: Chroma vector DB
- docs_sp.joblib: Preprocessed docs for BM25
- uniprot_sprot.fasta and related files: Sequence data
- queryfields.txt, result-fields.json, search-fields.json: Field definitions
- backend/
main.py: FastAPI serversrc/: Core logic (prompting, retrieval, embedding, etc.)asset/: All data files, embeddings, and vector DBsmodeldir/: HuggingFace model cache
- frontend/
src/components/: React UI componentssrc/pages/: Page-level React componentssrc/services/: API service modulespublic/: Static files (HTML, icons, manifest)
- config/: Setup and environment scripts
- test/: Test, evaluation, and visualization scripts
- vector_search_monitoring/, memory_monitoring/: Monitoring scripts and logs
ProQuest is an advanced, modular platform designed to make the vast knowledge in the UniProt protein database accessible through natural language. As biological databases grow in size and complexity, extracting meaningful insights—especially for queries involving sets of proteins or complex biological relationships—becomes increasingly challenging. ProQuest addresses this by combining large language models (LLMs) and retrieval-augmented generation (RAG) to enable intuitive, accurate, and evidence-grounded exploration of protein data.
Key Features:
- Natural Language Querying: Users can ask complex biological questions in plain language. ProQuest translates these into structured database queries, removing the need for specialized syntax or deep database expertise.
- Hybrid Retrieval Pipeline: The system integrates multiple retrieval strategies—semantic vector search (using protein language model embeddings), keyword-based BM25 ranking, full-text substring search, and sequence similarity search—ensuring that answers are both comprehensive and biologically relevant.
- Evidence-Grounded Answers: ProQuest’s RAG module synthesizes information from curated UniProt records, providing responses that are not only accurate but also traceable to their biological sources, minimizing hallucination and maximizing scientific reliability.
- Scalable and Extensible: The architecture is modular, supporting new retrieval strategies and easy adaptation to other biological databases.
System Components:
- ProteinSearch: Converts natural language queries into UniProt-compatible Solr queries using schema-informed prompt engineering and multi-model LLM sampling.
- SeqSim: Performs fast, biologically meaningful protein sequence similarity searches using ProtT5 embeddings and Annoy-based nearest-neighbor indexing.
- ProteinChat: Aggregates evidence from all retrieval modules and generates natural language answers, grounded in curated protein knowledge.
Why ProQuest? Traditional database interfaces require users to know both the biological context and the technical details of query languages. ProQuest lowers this barrier, making it possible for researchers of all backgrounds to access, interpret, and integrate protein knowledge at scale. Systematic evaluation shows that ProQuest’s hybrid retrieval and RAG approach delivers more accurate, complete, and biologically correct answers than LLMs or database queries alone.
For advanced usage, see individual scripts in config/ and test/.
For troubleshooting, check logs and monitoring outputs in the respective folders.