VideoLM: Unified RAG Knowledge Companion

VideoLM implements a complete Retrieval-Augmented Generation (RAG) pipeline designed to extract, summarize, and enable conversational Q&A over video content from YouTube.

The application processes a video from start to finish, turning raw transcription into a structured, searchable knowledge base. It uses specialized LLM components—including Mini Models for cue detection and a Smart Summary model—to deliver highly accurate, context-aware answers.

---

🌟 Features

• Full Pipeline Orchestration: Manages the entire ingestion flow from video download to final Q&A readiness.
• Whisper Transcription: Uses the Whisper model to generate time-stamped transcription.
• Mini Model Pre-Analysis: Identifies key focus points and extracts visual cue lines (moments where the instructor points to a slide or board).
• Context-Aware Summarization: The Smart Summary model generates intelligent summaries using RAG to retrieve relevant transcription chunks.
• Conversational Q&A (Smart Answer): Indexes content and chat history in ChromaDB to provide accurate, multi-turn answers, leveraging timestamp context.
• Streamlit UI: Simple, interactive interface for monitoring ingestion and chatting with the AI.

---

🚀 Quick Start Guide

1. Prerequisites

You must have Python 3.8+ and the following external dependency installed:

Dependency	Purpose	Installation Guide
FFmpeg	Required by yt-dlp and Whisper to process and extract audio from video streams.	Download Page (Official)
Gemini API Key	Required for all LLM-powered RAG components.	Get API Key (Google AI Studio)

---

2. Installation

Clone the repository:

git clone https://github.com/your-username/VideoLM-RAG.git
cd VideoLM-RAG

Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows, use: .\venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

---

3. Running the Application

Start the Streamlit application:

streamlit run app.py

Then, open the browser and:

1. Enter your Gemini API Key and a YouTube video URL (short educational videos recommended).
2. Click "Run Ingestion Pipeline (FULL FLOW)".
3. Wait for the four steps to complete.
4. Start asking questions in the Interactive Q&A chat.

---

💡 Example Questions

• "What are the main concepts covered in the video?"
• "Explain the backpropagation process mentioned."
• "What happens at 01:15?" (tests timestamp + visual cue context)

---

🏗 Architecture & Files

File	Component	Role
app.py	Orchestrator	Streamlit UI, initializes RAG components, and manages ingestion/Q&A flow.
video_processor.py	Media Pipeline	Handles URL parsing, yt-dlp audio download, and Whisper transcription.
rag_mini_models.py	Pre-Analysis Layer	Extracts timestamps, focus points, and visual cues using LLM prompts.
rag_smart_summary.py	Summary RAG	Indexes transcription, retrieves relevant chunks, and generates summaries.
rag_smart_answer.py	Conversational RAG	Manages ChromaDB indexing, embeddings, and context retrieval for Q&A.
requirements.txt	Dependencies	Lists required libraries (google-genai, chromadb, streamlit, etc.).

---

🧠 Future Enhancements

• Multilingual transcription and summarization support.
• Visual cue detection through frame analysis.
• Integration with external knowledge bases (Wikipedia, PDFs, Docs).
• User session persistence and dashboard analytics.

---

Developed for Hackathons | Built with ❤️ using Streamlit, Gemini API, and ChromaDB.