multimodal.md

Multimodal PDF RAG Pipeline

A lightweight, end-to-end Python implementation of a Multimodal Retrieval-Augmented Generation (RAG) pipeline using OpenAI's CLIP model for cross-modal embeddings and Google's Gemini 2.5 Flash for visually grounded synthesis.

This system extracts both text and embedded images (such as charts, tables, and graphs) from complex documents, projects them into a unified vector space, and answers semantic queries by processing text and visual context concurrently.

---

Architecture Overview

1. Extraction: Parses PDF pages using PyMuPDF (fitz), separating text data from embedded images (stored as raw bytes and encoded to base64 strings).
2. Cross-Modal Embedding: Uses a pre-trained CLIPModel (openai/clip-vit-base-patch32) to generate text vectors (from document chunks and user queries) and image vectors (from extracted visuals) within a shared geometric workspace.
3. Dimensional Realignment: Standardizes variable cross-modal embedding dimensions down to a clean, uniform 512-dimensional array slicing layout.
4. Vector Database Indexing: Loads aligned embeddings into a high-performance FAISS Inner Product (IndexFlatIP) similarity index for low-latency top-$k$ retrieval.
5. Synthesis Engine: Assembles a structured multi-element HumanMessage pairing text snippets and base64 images together, allowing Gemini 2.5 Flash via LangChain to generate visually grounded answers.

---

Technical Stack

• Orchestration: LangChain & langchain-google-genai
• Generative LLM: Google Gemini 2.5 Flash
• Vision-Language Embedding Model: OpenAI CLIP (ViT-B/32 via Hugging Face transformers)
• Vector Index Store: FAISS (Facebook AI Similarity Search)
• Document Processing: PyMuPDF (fitz) & Pillow (PIL)

---

Getting Started

Prerequisites

Ensure you have your Google Gemini API key exported or available in a local .env file:

GOOGLE_API_KEY=your_gemini_api_key_here