rag-document-qa

I got tired of hallucinated answers from LLMs when I asked about specific documents or topics they weren't trained on. RAG (Retrieval Augmented Generation) is the fix — instead of asking the model to remember things it doesn't know, you give it the relevant context at query time and it answers from that.

This project is a full RAG pipeline. Drop in your own .txt files, ask questions in plain English, and get answers with citations so you can verify them.

How it works

1. Your files get split into chunks
2. Each chunk gets embedded (converted to a vector that captures its meaning)
3. Your question also gets embedded
4. The most similar chunks are fetched
5. Those chunks + your question go to the LLM, which answers strictly from the context

Simple idea, but genuinely useful. The source attribution part is what I care about most — if the answer isn't in your documents, it says so.

Stack

• Python + LangChain 1.2
• Hybrid Support (Cloud or 100% Local)
  • Cloud Mode: OpenAI text-embedding-3-small (embeddings) + Groq llama-3.3-70b-versatile (generation)
  • Local Mode: Ollama nomic-embed-text (embeddings) + Ollama qwen2.5:1.5b (generation)
• FAISS for the local vector store (no external DB needed)

Project structure

rag-document-qa/
├── ingestion/loader.py       # loads and chunks your documents
├── embeddings/embedder.py    # creates embeddings + builds the vector store
├── retrieval/retriever.py    # semantic search + answer generation
├── data/                     # drop your .txt files here
└── main.py

Setup

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Configuration

By default, the system runs 100% Locally using Ollama to save API costs and protect privacy.

1. Download Ollama
2. Pull the required models: ollama pull nomic-embed-text and ollama pull qwen2.5:1.5b
3. Ensure USE_LOCAL_MODELS=True is in your .env.

If you prefer to use Cloud Models, set USE_LOCAL_MODELS=False and provide your keys:

• GROQ_API_KEY
• OPENAI_API_KEY

Add your .txt or .md files to the data/ folder, then:

python main.py

There's already a sample file in data/ so you can test it right away without adding anything.

Notes

• Groq handles answer generation (free)
• OpenAI handles embeddings — this is the only paid part, and it's tiny (fractions of a cent per document)
• FAISS stores everything in memory, no external services needed
• Chunking uses overlap so context doesn't get cut off at boundaries