A production-grade, streaming-first architecture for real-time ride analytics. Simulates 100+ events/sec with geospatial processing, surge detection, and live dashboards.
Producers (Python Faker)
β
Apache Kafka (event stream)
β
PySpark Structured Streaming (windowed aggregations, geofencing)
β
Dual Storage:
βββ Redis (hot metrics, <1ms lookups)
βββ S3 + Iceberg (cold storage, historical analysis)
βββ PostgreSQL + PostGIS (geospatial queries)
β
Streamlit Dashboard (live visualization)
β
100+ Events/Second: Simulates realistic NYC ride patterns
β
Sub-Second Latency: Redis caching for instant metric lookups
β
Geospatial Processing: Zone-based surge detection
β
Windowed Aggregation: 5-minute windows with watermarking
β
Late Data Handling: 2-minute grace period for delayed events
β
Data Quality: Event and aggregation integrity validators
β
Infrastructure as Code: Production-hardened Terraform with IAM instance profiles & S3 public access blocks
β
Production Containerization: Non-root, multi-stage Docker builds with .dockerignore context isolation
β
Automated CI/CD: GitHub Actions pipeline with Black, Flake8, Pytest, Terraform validation, and Trivy CVE vulnerability scans
- Surge Pricing: Dynamic multiplier based on demand/supply
- Request Volume: 5-minute rolling counts by zone
- Active Drivers: Real-time driver distribution
- Completion Rate: Ride success metrics
- ETA Accuracy: Trip distance and duration
- Docker & Docker Compose
- Python 3.11+ (with Java 17 installed on the host for Spark)
- AWS CLI (for Terraform deployment)
The easiest and most efficient way to run the entire pipeline locally is using the provided Makefile targets:
# 1. Install Python requirements on the host
make install
# 2. Start the Docker Compose infrastructure (Kafka, Postgres, Redis, Streamlit, etc.)
make start
# 3. Start the Spark Stream Processor on the host (cleans checkpoints and sets host network overrides)
make local-processor
# 4. Access the dashboard at http://localhost:8501# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Start Kafka locally (or via Docker)
# Start services in separate terminals:
# Terminal 1: Kafka Producer
python producer/kafka_producer.py
# Terminal 2: Spark Streaming (requires Spark 4.1.1 & Java 17)
JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 spark-submit \
--packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.1.1 \
processor/spark_stream_processor.py
# Terminal 3: Streamlit Dashboard
streamlit run serving/streamlit_app.pyReal_Time_Ride-Analysis/
βββ producer/
β βββ kafka_producer.py # Generates 100+ events/sec
βββ processor/
β βββ spark_stream_processor.py # Streaming aggregations
βββ serving/
β βββ streamlit_app.py # Live dashboard
β βββ redis_handler.py # Cache operations
βββ config/
β βββ schemas.py # Avro event schemas
β βββ settings.py # Config management
βββ data_quality/
β βββ validators.py # Data integrity checks
βββ terraform/
β βββ main.tf # AWS infrastructure
β βββ variables.tf # Terraform variables
β βββ user_data.sh # EC2 initialization
βββ docker-compose.yml # Local stack
βββ Dockerfile.producer # Producer container
βββ Dockerfile.streamlit # Dashboard container
βββ requirements.txt # Python dependencies
βββ .env.example # Environment template
βββ README.md # This file
Copy .env.example to .env and update values:
cp .env.example .envKey variables:
KAFKA_BOOTSTRAP_SERVERS: Kafka connectionNUM_DRIVERS: Number of simulated drivers (default: 50)EVENTS_PER_SECOND: Event generation rate (default: 100)SURGE_THRESHOLD: Requests triggering surge (default: 100)WINDOW_DURATION_MINUTES: Aggregation window (default: 5)
The pipeline includes Great Expectations validators:
from data_quality.validators import event_validator
event = {...}
is_valid, errors = event_validator.validate_event(event)
if not is_valid:
print(f"Validation errors: {errors}")Checks performed:
- Required fields presence
- Latitude/longitude bounds (-90/+90, -180/+180)
- Trip distance validity (0-5000 km)
- Surge multiplier range (0.5-5.0x)
- Timestamp freshness
cd terraform
# Initialize Terraform
terraform init
# Plan deployment
terraform plan -var="aws_region=us-east-1"
# Apply infrastructure
terraform apply -var="aws_region=us-east-1"Resources created:
- S3 bucket for Iceberg data lake
- EC2 instance for Spark processor
- CloudWatch logs for monitoring
- IAM roles & policies
The dashboard provides:
- Real-time Metrics: Requests, drivers, surge status by zone
- Demand Charts: Bar charts showing distribution
- Surge Alerts: Notifications when thresholds exceeded
- Cache Stats: Redis performance monitoring
- System Config: Current settings and parameters
Access at: http://localhost:8501
docker compose exec kafka kafka-console-consumer \
--bootstrap-server localhost:9092 \
--topic ride_events \
--from-beginningdocker compose exec redis redis-cli
> KEYS surge:*
> GET surge:manhattan:currentSpark UI available at: http://localhost:4040
docker compose exec postgres psql -U postgres -d ride_analyticsThis project demonstrates:
- Stream Processing: Kafka, PySpark Structured Streaming
- Geospatial: PostGIS, zone-based aggregations
- Real-time Systems: Windowing, watermarking, late data handling
- Data Quality: Great Expectations
- Infrastructure: Docker, Terraform, AWS
- Data Visualization: Streamlit, Plotly
- Cache Optimization: Redis for sub-ms lookups
- Error:
KeeperException$NodeExistsException: KeeperErrorCode = NodeExists - Solution: This happens on quick restarts when ZooKeeper has not expired the old broker session. Stop the stack completely and restart it:
make stop make start
- Error:
FileNotFoundException: File file:/tmp/ride_analytics_checkpoint_csv/state - Solution: The Spark checkpoint state is out-of-sync. Clean the temporary files and restart the processor:
make local-processor
- Ensure Kafka container is healthy:
docker compose ps - Check bootstrap servers:
docker compose logs kafka
- Verify Redis connection:
docker compose logs streamlit - Check producer is running:
docker compose logs producer
- Reduce
NUM_DRIVERSin.env - Reduce
EVENTS_PER_SECOND - Increase Docker memory limits
- Implement schema registry for Avro
- Add Iceberg table format on S3
- Set up Airflow/Dagster for orchestration
- Implement alerting (PagerDuty, Slack)
- Add authentication to Streamlit
- Set up CI/CD with GitHub Actions
- Implement dbt transformations for Gold layer
- Add distributed tracing (Jaeger)
MIT
Built for showcase in Data Science & AI portfolio.
Questions? Check the documentation or open an issue!