A production-ready, scalable FAQ generation system that scrapes social media data (Reddit), processes it with Apache Spark, and generates intelligent FAQs using LLMs (Google Gemini / Ollama).
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β FAQ Generation System β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β ββββββββββββββββ βββββββββββββββ ββββββββββββββββ β
β β Reddit ββββββΆβ Apache ββββββΆβ HDFS β β
β β Scraper(Go) β β Kafka β β Storage β β
β ββββββββββββββββ βββββββββββββββ ββββββββββββββββ β
β β β β
β β βΌ β
β β ββββββββββββββββ β
β β β Apache β β
β β β Spark β β
β β ββββββββββββββββ β
β β β β
β βΌ βΌ β
β ββββββββββββββββ ββββββββββββββββ β
β β MongoDB ββββββ FAQ β β
β β β β Generator β β
β ββββββββββββββββ β (Gemini/ β β
β β β Ollama) β β
β β ββββββββββββββββ β
β βΌ β
β ββββββββββββββββ β
β β FastAPI β β
β β Backend β β
β ββββββββββββββββ β
β β β
β βΌ β
β ββββββββββββββββ β
β β React β β
β β Dashboard β β
β ββββββββββββββββ β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
- Multi-Source Scraping: Go-based Reddit scraper with rate limiting (100 posts/minute)
- Real-time Streaming: Apache Kafka for reliable message queuing
- Distributed Storage: HDFS for storing raw and processed data
- Big Data Processing: Apache Spark for data cleaning and analysis
- AI-Powered FAQ Generation: Google Gemini with Ollama fallback
- Modern Dashboard: React/Next.js UI for product management
- REST API: FastAPI backend for all operations
- Fully Dockerized: Production-ready Docker Compose setup
- Docker & Docker Compose
- 8GB+ RAM (recommended 16GB for full stack)
- Reddit API credentials
- Google Gemini API key (optional, has Ollama fallback)
- Ollama installed locally (optional, for local LLM)
cd faq_generation
# Copy environment template
cp .env.example .env
# Edit .env with your credentials
nano .envEdit .env file:
# Reddit API Credentials
REDDIT_CLIENT_ID=your_reddit_client_id
REDDIT_CLIENT_SECRET=your_reddit_client_secret
REDDIT_USER_AGENT=CompanyFAQScraper/1.0 by u/your_username
# LLM Configuration
GEMINI_API_KEY=your_gemini_api_key
OLLAMA_HOST=http://ollama:11434
OLLAMA_MODEL=llama3.2
USE_OLLAMA_FALLBACK=trueFor Development (lighter weight):
docker-compose -f docker-compose.dev.yml up -dFor Production (full stack):
docker-compose up -d| Service | URL | Description |
|---|---|---|
| Dashboard | http://localhost:3000 | Web UI for managing products and FAQs |
| API | http://localhost:8000 | REST API endpoints |
| API Docs | http://localhost:8000/docs | Swagger documentation |
| Kafka UI | http://localhost:8080 | Kafka monitoring |
| MongoDB UI | http://localhost:8081 | Database browser |
| HDFS UI | http://localhost:9870 | Hadoop file browser |
| Spark UI | http://localhost:8082 | Spark monitoring |
- Open the Dashboard at http://localhost:3000
- Click "Add Product"
- Enter product details:
- Name: e.g., "Samsung Galaxy S24"
- Subreddits: technology, gadgets, samsung, samsunggalaxy
- Keywords: review, problem, help, question, experience
- Select a product from the list
- Click "Run Pipeline" (
βΆοΈ ) to:- Scrape Reddit for discussions
- Store data in HDFS via Kafka
- Generate FAQs using AI
# Health check
curl http://localhost:8000/health
# List products
curl http://localhost:8000/api/products
# Add a product
curl -X POST http://localhost:8000/api/products \
-H "Content-Type: application/json" \
-d '{"name": "Samsung Galaxy S24", "subreddits": ["samsung", "gadgets"]}'
# Trigger scraping
curl -X POST http://localhost:8000/api/scrape \
-H "Content-Type: application/json" \
-d '{"product_id": "your_product_id"}'
# Generate FAQs
curl -X POST http://localhost:8000/api/generate-faqs \
-H "Content-Type: application/json" \
-d '{"product_id": "your_product_id"}'
# Get FAQs
curl http://localhost:8000/api/faqs/Samsung%20Galaxy%20S24- OAuth2 authentication with Reddit API
- Rate-limited requests (configurable, default 100/min)
- Produces messages to Kafka
- HTTP endpoint for triggering scrapes
- Consumes raw social data from Kafka
- Batches messages for efficiency
- Stores data to HDFS in JSON format
- Cleans and normalizes text data
- Extracts questions from content
- Calculates relevance scores
- Clusters similar content
- Saves to MongoDB for FAQ generation
- Supports Google Gemini API
- Ollama fallback for local LLM
- Generates structured FAQs with:
- Questions and answers
- Categories
- Confidence scores
- Source references
- Keywords
- RESTful API for all operations
- Product management CRUD
- Pipeline orchestration
- FAQ retrieval
- Next.js 14 with TypeScript
- Tailwind CSS styling
- Real-time status updates
- FAQ search and filtering
| Variable | Default | Description |
|---|---|---|
SCRAPE_RATE_LIMIT |
100 | Requests per minute |
SCRAPE_INTERVAL_SECONDS |
60 | Interval between scrape cycles |
| Variable | Default | Description |
|---|---|---|
KAFKA_TOPIC_RAW_DATA |
raw_social_data | Topic for scraped data |
KAFKA_CONSUMER_GROUP |
faq_processor | Consumer group ID |
| Variable | Default | Description |
|---|---|---|
GEMINI_API_KEY |
- | Google Gemini API key |
OLLAMA_HOST |
http://ollama:11434 | Ollama server URL |
OLLAMA_MODEL |
llama3.2 | Ollama model to use |
USE_OLLAMA_FALLBACK |
true | Fall back to Ollama if Gemini fails |
- Access Kafka UI at http://localhost:8080
- Monitor topics, consumer groups, and message flow
- Access HDFS UI at http://localhost:9870
- Browse stored data at
/data/raw/and/data/processed/
- Access Mongo Express at http://localhost:8081
- Credentials: admin / admin123
- Browse
faq_generationdatabase
# Check logs
docker-compose logs -f [service_name]
# Restart specific service
docker-compose restart [service_name]# Check Kafka health
docker-compose exec kafka kafka-broker-api-versions --bootstrap-server localhost:9092# Check HDFS status
docker-compose exec namenode hdfs dfsadmin -report# Pull the model first
ollama pull llama3.2
# Test Ollama
curl http://localhost:11434/api/tags- Add more Kafka partitions for parallel processing
- Add Spark workers for distributed processing
- Deploy multiple API instances behind a load balancer
- Use managed Kafka (Confluent Cloud, AWS MSK)
- Use managed MongoDB (Atlas)
- Use managed HDFS (AWS EMR, GCP Dataproc)
- Implement proper authentication and authorization
- Set up monitoring with Prometheus/Grafana
faq_generation/
βββ docker-compose.yml # Production compose file
βββ docker-compose.dev.yml # Development compose file
βββ .env.example # Environment template
βββ hadoop/
β βββ hadoop.env # Hadoop configuration
βββ scrapers/
β βββ reddit-scraper/ # Go Reddit scraper
β βββ main.go
β βββ go.mod
β βββ Dockerfile
βββ services/
β βββ kafka-consumer/ # Kafka to HDFS consumer
β β βββ consumer.py
β β βββ requirements.txt
β β βββ Dockerfile
β βββ spark-processor/ # Spark data processor
β β βββ processor.py
β β βββ requirements.txt
β βββ faq-generator/ # LLM FAQ generator
β β βββ generator.py
β β βββ requirements.txt
β β βββ Dockerfile
β βββ api/ # FastAPI backend
β βββ main.py
β βββ requirements.txt
β βββ Dockerfile
βββ dashboard/ # React/Next.js frontend
βββ app/
βββ package.json
βββ Dockerfile
MIT License
- Fork the repository
- Create a feature branch
- Make your changes
- Submit a pull request