Distributed Hybrid RAG Engine

A production-grade, distributed Retrieval-Augmented Generation (RAG) platform built with FastAPI, Celery, PostgreSQL (pgvector + tsvector), FlashRank, and Groq LPUs.

This system addresses common bottlenecks in naive RAG pipelines, such as event-loop starvation during parsing, semantic blind spots in pure dense retrieval, and bi-encoder scoring inaccuracy. It combines asynchronous ingestion, two-stage hybrid search with Reciprocal Rank Fusion (RRF), deep cross-encoder re-ranking, and grounded generation with chunk-level attribution.

---

Table of Contents

• Architecture Overview
• Key Engineering Highlights
• Tech Stack
• Project Structure
• Local Setup
• API Reference
• Architectural Decisions & Trade-Offs
• License

---

Architecture Overview

🔗 View the Architecture Diagram

---

Key Engineering Highlights

1. Asynchronous Event-Driven Ingestion

• Non-Blocking Ingestion: Document uploads return an immediate HTTP 202 Accepted response with an ingestion tracking UUID.
• Worker Isolation: Heavy PDF text extraction, sliding-window chunking (chunk_size=300, chunk_overlap=50), and batch embedding computations are offloaded to Celery workers over a Redis message queue.
• Failure Safety: Database updates run within explicit session boundaries, rolling back cleanly and marking the document FAILED if errors occur mid-stream.

2. Multi-Modal Two-Stage Retrieval

• Branch A (Dense Semantic Retrieval): Uses pgvector to run cosine distance scans (<=>) over 384-dimensional dense vectors generated by all-MiniLM-L6-v2.

• Branch B (Sparse Lexical Retrieval): Uses PostgreSQL native tsvector with ts_rank_cd to execute cover-density keyword matching for exact IDs, technical terms, and code snippets.

• Reciprocal Rank Fusion (RRF): Merges dense and sparse outputs using positional rankings rather than unstable, uncalibrated score weights:

  $$RRF_Score(d) = \sum_{m \in {\text{dense}, \text{sparse}}} \frac{1}{k + \text{rank}_m(d)} \quad (k = 60)$$

• Cross-Encoder Re-Ranking: The top 10 RRF candidates pass through flashrank (ms-marco-MiniLM-L-12-v2). The model scores full query-document token interactions simultaneously via self-attention, pruning false positives before the context hits the LLM.

3. Sub-Second Grounded Synthesis

• Ultra-Low Latency: Feeds the top 3 cross-encoder re-ranked chunks to Groq's LPU infrastructure running llama-3.3-70b-versatile.
• Strict Context Boundary: Enforces a system prompt that mandates direct source citations ([Source Chunk X]) and suppresses hallucinations.

---

Tech Stack

Domain	Technology	Purpose
API Runtime	Python 3.11+, FastAPI, Uvicorn	Asynchronous web interface with dependency-injected database sessions
Task Queue	Celery 5, Redis 7	Distributed worker orchestration and task serialization
Database & Search	PostgreSQL 16, pgvector, Full-Text Search	Unified relational storage, vector indices, and inverted lexical search
Embeddings & Re-ranking	sentence-transformers (all-MiniLM-L6-v2), flashrank	Bi-encoder vector representations and cross-encoder attention re-ranking
LLM Inference	Groq Cloud API (llama-3.3-70b-versatile)	Fast, grounded answer synthesis
Frontend	React 18, Vite, Lucide Icons, React-Markdown	Single-page UI with real-time status polling, metric visualization, and Markdown support

---

Project Structure

Hybrid_RAG_Engine/
├── backend/
│   ├── app/
│   │   ├── core/
│   │   │   ├── config.py             # Pydantic BaseSettings environment parsing
│   │   │   └── database.py           # Async (asyncpg) and Sync (psycopg) database session pools
│   │   ├── models/
│   │   │   └── schema.py             # SQLAlchemy ORM models (Document, DocumentChunk)
│   │   ├── services/
│   │   │   ├── embedding.py          # embedding service
│   │   │   ├── generator.py          # Groq LLM synthesis client with strict prompt constraints
│   │   │   ├── parser.py             # PyPDF text extraction and sliding-window chunker
│   │   │   ├── reranker.py           # FlashRank cross-encoder service
│   │   │   └── search.py             # Hybrid search engine (pgvector + tsvector + RRF)
│   │   ├── workers/
│   │   │   └── celery_worker.py      # Background ingestion pipeline task
│   │   └── main.py                   # FastAPI application with modern lifespan management
│   ├── Dockerfile                    # Container configuration
│   ├── entrypoint.sh                 # Container entrypoint orchestrator
│   └── requirements.txt              # Pinned Python package dependencies
├── frontend/
│   ├── src/
│   │   ├── App.jsx                   # React dashboard with live polling and search
│   │   ├── index.css                 # Dark-mode responsive styling
│   │   └── main.jsx                  # React application entry point
│   ├── package.json                  # Node.js dependencies
│   └── vite.config.js                # Vite build and dev-server configuration
├── docker-compose.yml                # Local infrastructure setup (PostgreSQL 16 + Redis 7)
└── README.md                         # System documentation

---

Local Setup

1. Prerequisites

Make sure the following are installed:

• Docker & Docker Compose
• Python 3.11+
• Node.js v18+ and npm
• A free Groq Cloud API key

2. Infrastructure Setup (Docker Compose)

Start the PostgreSQL container (with pgvector enabled) and the Redis broker from the project root:

docker compose up -d

Verify the containers are running:

docker ps

Service	Address
PostgreSQL	localhost:5435
Redis	localhost:6379

3. Backend Setup

a. Create and activate a virtual environment

cd backend
python -m venv .venv
source .venv/bin/activate    # On Windows: .venv\Scripts\activate

b. Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

c. Configure environment variables

Create a .env file in the backend/ directory:

# PostgreSQL Connection Strings
DATABASE_URL=postgresql+asyncpg://rag_user:rag_password@localhost:5435/rag_engine
SYNC_DATABASE_URL=postgresql+psycopg://rag_user:rag_password@localhost:5435/rag_engine

# Redis Configuration
REDIS_URL=redis://localhost:6379/0

# Vector Settings
VECTOR_DIMENSION=384

# Groq Cloud API
GROQ_API_KEY=gsk_your_actual_groq_api_key_here
GROQ_MODEL=llama-3.3-70b-versatile

d. Start the Celery background worker

Open a dedicated terminal, navigate to backend/, activate your .venv, and run:

celery -A app.workers.celery_worker.celery_app worker --loglevel=info

e. Start the FastAPI gateway

Open a separate terminal, navigate to backend/, activate your .venv, and run:

uvicorn app.main:app --reload --port 8000

Interactive API documentation is available at http://localhost:8000/docs.

4. Frontend Setup

a. Install dependencies

cd frontend
npm install

b. Configure environment variables

Create a .env file in the frontend/ directory:

VITE_API_BASE=http://localhost:8000/api/v1

c. Start the Vite development server

npm run dev

Open your browser at http://localhost:5173.

Tip: You need three processes running at the same time: the Celery worker, the FastAPI server, and the Vite dev server. Docker must also be up for PostgreSQL and Redis.

---

API Reference

Base URL: http://localhost:8000/api/v1

1. Document Upload (Asynchronous)

Endpoint	POST /api/v1/documents/upload
Content-Type	multipart/form-data
Response Status	202 Accepted

Response

{
  "document_id": "c7a4b123-5e82-4f3b-b27e-8531d044f51a",
  "filename": "microservices_architecture.pdf",
  "status": "PENDING",
  "message": "Document accepted for asynchronous background processing."
}

2. Ingestion Status Polling

Endpoint	GET /api/v1/documents/{document_id}/status
Response Status	200 OK

Response

{
  "document_id": "c7a4b123-5e82-4f3b-b27e-8531d044f51a",
  "filename": "microservices_architecture.pdf",
  "status": "COMPLETED",
  "total_chunks": 42,
  "created_at": "2026-10-02T18:32:00.124510"
}

3. Two-Stage RAG Query with Re-ranking & Synthesis

Endpoint	POST /api/v1/query
Content-Type	application/json

Request Payload

{
  "query": "How is the circuit breaker pattern implemented in distributed systems?",
  "top_candidates": 10,
  "final_top_k": 3
}

Response Payload

{
  "query": "How is the circuit breaker pattern implemented in distributed systems?",
  "answer": "The circuit breaker pattern wraps calls to remote services to prevent cascading failures [Source Chunk 1]. When the failure threshold is exceeded, the circuit trips to an OPEN state, failing fast without hitting the degraded dependency [Source Chunk 2]...",
  "sources": [
    {
      "chunk_id": "89b70b55-d365-4f4b-84a1-b8efd806a38f",
      "document_id": "c7a4b123-5e82-4f3b-b27e-8531d044f51a",
      "content": "A circuit breaker monitors for failures. Once failures reach a predefined threshold, the breaker trips...",
      "chunk_index": 12,
      "rrf_score": 0.032522,
      "dense_rank": 2,
      "sparse_rank": 1,
      "cross_encoder_score": 0.9412
    }
  ]
}

---

Architectural Decisions & Trade-Offs

1. PostgreSQL + pgvector vs. Standalone Vector Databases

• Atomic Consistency: Using PostgreSQL as both the relational and vector store allows cascading deletes (ON DELETE CASCADE). Deleting a document atomically purges its metadata, raw text, embeddings, and inverted indexes inside a single ACID transaction, preventing orphaned vectors.
• Operational Simplicity: Avoids managing external synchronization pipelines, network overhead, and separate access controls for external vector databases (e.g., Pinecone or Milvus).

2. Reciprocal Rank Fusion (RRF) vs. Linear Score Blending

• Unbounded Score Variance: Cosine distance is strictly bounded ($[0, 2]$), whereas BM25 / ts_rank_cd scores are unbounded ($[0, \infty)$) and change based on document collection size.
• Scale-Invariance: RRF scores candidates purely by their relative rank order, making it stable across diverse document types without manual weight tuning.

3. Two-Stage Retrieval (Bi-Encoder + Cross-Encoder)

• Performance: Running cross-attention across thousands of chunks introduces severe latency bottlenecks. A bi-encoder (all-MiniLM-L6-v2) handles initial candidate retrieval across the entire database in under 20 ms, while the cross-encoder (flashrank) evaluates only the top 10 candidates with full token-to-token attention.

---

License

This project is licensed under the MIT License.