🎛️ Video Assistant

Turn any video or audio into a transcript, summary, action items, and a chatbot — powered by Whisper, LangChain, Groq & RAG

---

📌 Overview

Video Assistant is an end-to-end GenAI application that takes a YouTube URL or a local video/audio file and converts it into structured, searchable knowledge.

It downloads and chunks the audio, transcribes it (locally with Whisper, or via Sarvam AI for Hinglish), then uses LLMs on Groq to generate a title, summary, action items, key decisions, and open questions. Finally, it indexes the transcript in a vector database so you can chat with your video using Retrieval-Augmented Generation (RAG).

It works on any spoken content — lectures, interviews, podcasts, tutorials, meetings, and more.

✨ Features

• 🎥 Flexible input — YouTube/URL via yt-dlp, or upload mp4, mp3, wav, m4a, mov, webm
• 🗣️ Dual transcription engine
  • English → OpenAI Whisper (runs locally, no API cost)
  • Hinglish → Sarvam AI (transcribes and translates to English)
• 📝 Map-reduce summarization — long transcripts are split, summarized per chunk, then merged into a final bullet-point summary
• 🧠 Structured extraction — action items (task / owner / deadline), key decisions, and open questions
• 💬 Chat with your video — RAG pipeline with Chroma + HuggingFace embeddings, answers grounded only in the transcript
• 🖥️ Two interfaces — a polished dark "Control Room" Streamlit UI and a CLI
• ⬇️ Export — download the transcript as .txt

🏗️ Architecture

 YouTube URL / File
        │
        ▼
 ┌──────────────┐   yt-dlp + pydub + FFmpeg
 │ Audio Chunker│
 └──────┬───────┘
        ▼
 ┌──────────────┐   english  → Whisper (local)
 │ Transcriber  │   hinglish → Sarvam AI (cloud)
 └──────┬───────┘
        ▼
   Full Transcript
        │
        ├──► Title Generator ─────────┐
        ├──► Map-Reduce Summarizer ───┤
        ├──► Action Item Extractor ───┼──► Streamlit UI / CLI
        ├──► Decision Extractor ──────┤
        ├──► Open Question Extractor ─┘
        │
        └──► Chunk (500 chars) ─► MiniLM embeddings ─► Chroma
                                                          │
                                  Question ─► Retriever (top-4) ─► Groq LLM ─► Answer

🧰 Tech Stack

Layer	Technology
Media acquisition	yt-dlp, pydub, ffmpeg-python
Speech-to-text	OpenAI Whisper (local), Sarvam AI (saaras:v3)
LLM orchestration	LangChain (LCEL), langchain-groq
LLM	openai/gpt-oss-120b served on Groq
Embeddings	sentence-transformers/all-MiniLM-L6-v2
Vector store	ChromaDB
UI	Streamlit (custom CSS theme)

📁 Project Structure

ai-video-assistant-rag/
├── app.py                    # Streamlit UI ("Control Room" theme)
├── Main.py                   # CLI entry point (pipeline + terminal chat)
├── core/
│   ├── transcriber.py        # Whisper / Sarvam routing and chunk transcription
│   ├── summarizer.py         # Title + map-reduce summary
│   ├── extractor.py          # Action items, decisions, open questions
│   ├── rag_engine.py         # RAG chain construction and Q&A
│   └── vector_store.py       # Chroma build / load / retriever
├── utilities/
│   └── audio_processor.py    # Download, convert, and chunk audio
├── requirements.txt
├── .env.example              # template for your API keys
├── .env                      # your real API keys (not committed)
└── README.md

🚀 Getting Started

Prerequisites

• Python 3.10+ (developed on 3.13)
• FFmpeg installed and on your PATH
• A Groq API key
• (Optional, for Hinglish) a Sarvam AI API key

Installation

# 1. Clone the repository
git clone https://github.com/<your-username>/ai-video-assistant-rag.git
cd ai-video-assistant-rag

# 2. Create and activate a virtual environment
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

Configuration

Copy .env.example to .env in the project root (next to app.py and Main.py) and add your keys:

GROQ_API_KEY=your_groq_api_key
SARVAM_API_KEY=your_sarvam_api_key      # only needed for Hinglish
SARVAM_STT_MODEL=saaras:v3
WHISPER_MODEL=base                      # tiny | base | small | medium | large

⚠️ Never commit your .env file. It is already listed in .gitignore.

Run the web app

streamlit run app.py

Run the CLI

python Main.py

You will be asked for a YouTube URL or file path and a language (english / hinglish). After processing, you can chat with the video in the terminal (exit to quit).

💡 How It Works

1. Chunk — the source is downloaded/converted and split into audio chunks.
2. Transcribe — each chunk is routed to Whisper (English) or Sarvam AI (Hinglish, split into 25-second pieces to respect API limits).
3. Summarize — the transcript is split into 3,000-character parts (200 overlap), each summarized, then merged into one professional summary.
4. Extract — three LCEL chains pull out action items, decisions, and open questions.
5. Index — the transcript is split into 500-character chunks, embedded with MiniLM, and stored in Chroma.
6. Chat — each question retrieves the top-4 relevant chunks, and the LLM answers strictly from that context (or says it couldn't find the answer).

🛣️ Roadmap

• Speaker diarization
• PDF export of the full report
• Timestamped transcript with jump-to-source citations in chat
• Support for more languages
• Docker image and one-click deployment

🤝 Contributing

Contributions, issues, and feature requests are welcome. Feel free to open an issue or submit a pull request.

👤 Author

Alfez Khan — Data Scientist | AI/ML & Generative AI

---

⭐ If you found this project useful, consider giving it a star!