A full-stack web application that provides real-time analytics and predictive insights for a real estate capital expenditure portfolio.
Note on Performance: The backend is deployed on Render's free tier, which spins down the service after a period of inactivity. As a result, the initial load may take up to 30 seconds while the server performs a "cold start". Subsequent requests will be much faster.
Project Lighthouse is a comprehensive, full-stack demonstration of a data analytics and intelligence solution for a fictional real estate investment firm. It showcases an end-to-end workflow, from data simulation and ETL to predictive modeling, model explainability, and a dynamic frontend dashboard.
The project is composed of a Python backend (data processing and API) and a React frontend.
The backend is a sophisticated data processing pipeline that feeds a Flask API.
-
1_generate_data.py: Simulates a realistic dataset of 200 capital expenditure projects usingpandasandFaker. It generates a rich set of features including project types, budgets, timelines, and vendor assignments. -
2_build_database.py: An advanced ETL (Extract, Transform, Load) script that:- Validates the raw CSV data against a formal schema using
panderato ensure data quality and integrity. - Transforms the data by cleaning it and engineering new features such as
ScheduleVariance_DaysandBudgetVariance_CAD. - Loads the data into a normalized SQLite database, splitting the information into
projects,vendors, andpropertiestables.
- Validates the raw CSV data against a formal schema using
-
3_enhanced_prediction_model.py: A script that demonstrates a complete MLOps workflow by training and applying three distinct models:- Risk Prediction: A
RandomForestClassifieris trained on completed projects to predict which ongoing projects are "At Risk" of being late or over budget. - Cost Prediction: A
RandomForestRegressorpredicts the finalActualCostof ongoing projects based on their features. - Duration Prediction: A second
RandomForestRegressorpredicts theActualDuration_Daysfor ongoing projects. - The script then updates the
projectstable with these predictions.
- Risk Prediction: A
-
4_app.py: AFlaskAPI that serves the enriched data from the database. It features a/api/projectsendpoint with dynamic filtering capabilities and is CORS-enabled to communicate with the frontend.
The application uses a normalized SQLite database to ensure data integrity and prevent redundancy. The schema is composed of three tables:
properties: Stores information about each property, including its name and city.vendors: Stores a list of unique vendors that can be assigned to projects.projects: The main table containing all project-specific information, including timelines, budgets, and the predictions from our models. It is linked to thepropertiesandvendorstables via foreign keys.
- Frontend: React, Chart.js, Bootstrap, Axios
- Backend: Python, Flask
- Database: SQLite
- Data Science & AI: Pandas, Scikit-learn, SHAP, Pandera
To get a local copy up and running, follow these simple steps.
- Python 3
- Node.js and npm
-
Backend Setup
# Clone the repo git clone https://github.com/Kevin-yyuan/project_lighthouse.git cd your_project # Create a virtual environment and activate it python3 -m venv venv source venv/bin/activate # Install Python dependencies pip install -r requirements.txt # Run the data pipeline and prediction models python 1_generate_data.py python 2_build_database.py python 3_enhanced_prediction_model.py # Start the backend server flask --app 4_app run
-
Frontend Setup
# In a new terminal, navigate to the frontend directory cd frontend # Install NPM packages npm install # Start the frontend development server npm start