Data Science Task

Welcome to the data-science-task repository. The project contains two main parts: the Data Science (DS) part and the Machine Learning Engineering (MLE) part. This project makes use of powerful tools and frameworks. Here are the key ones:

• 
• 
• 
• 
• 
• 

Getting Started

To get a local copy of this project and running on your computer, follow these simple steps.

Prerequisites

Before you begin, ensure you have git installed on your computer. To install git, download it from Git's official site.

Cloning the Project

1. Open your terminal: Launch your terminal application.

2. Choose a directory: Navigate to the directory where you want to clone the project using the cd command.

   cd path/to/your/directory

3. Clone the repository: Copy the project into your local machine by running the following command in your terminal. Make sure to replace with the actual URL of the repository you want to clone.

   git clone <repository-url>

4. Navigate to the project directory: Once the cloning process is complete, move into the project directory.

   cd repository-name

5. Check the files: You can list the files in the directory to see all the project files you have cloned.

   ls

Project Structure

This project has a modular structure, where each folder has a specific duty.

/data-science-task/
|-- data/                        
│   |--raw/
│   |--processed/
|-- notebooks/   
|   |--notebook.ipynb
|-- src/           
│   |--data_manipulation/
|   |  |--data_upload.py          
│   |  |--data_preprocess.py      
|   |  |--data_visualization.py  
|   |--train/                     
|   |  |--train.py 
|   |  |--Dockerfile
|   |  |--entrypoint.sh
|   |--inference/                
|   |  |--run_inference.py
|   |  |--Dockerfile
|   |  |--entrypoint.sh     
|   |-- main.py                  
|-- outputs/
|   |--models/
|   |  |--best_model.pkl
|   |--predictions/
|   |  |--predictions.csv
|   |  |--metric.txt    
|   | --figures/
|   |  |--plots.png
|   |--vectorizer/
|   |  |--tf_idf_vectorizer.pkl
|-- settings.json               
|-- requirements.txt            
|-- README.md

DS part

This part of the project focuses on exploring, modeling, and deriving insights from the dataset. It includes comprehensive steps from initial data analysis to feature engineering and model building. notebooks directory contains Jupyter notebook used for analysis and modeling.

Data

The dataset includes 50,000 movie reviews split into two parts: 40,000 reviews for training and 10,000 for testing. These reviews are taken from various movies and are used for binary sentiment classification, which means the model learns to classify each review as either positive or negative.

Workflow

A. Exploratory Data Analysis (EDA)

During the exploratory data analysis, we observed that the dataset consists of evenly balanced sentiments across the training and testing datasets, ensuring consistency between the two. This balance is crucial for unbiased model training.

Both datasets show right-skewed distributions indicating most movie reviews are relatively short, with a few extending much longer, evidenced by a long tail in histograms.

Word clouds reveal distinct language use in positive and negative reviews; words like 'great' and 'good' dominate positive reviews, while 'bad' and 'worst' appear frequently in negative ones.

The bar chart highlighting common words like 'movie', 'film', and 'like' underscores typical themes and sentiments.

In summary, test set is a good representative of the training set, which is beneficial for reliable model training.

B. Feature Engineering

To ensure our models perform optimally, we carried out several preprocessing steps on our text data:

• Tokenization: We started by breaking down each review into individual words or tokens. This is crucial for the analysis and manipulation of the text data.
• Character Filtering: We removed non-alphabetic characters from the text to standardize the data, and converted all characters to lowercase to prevent the same words in different cases from being treated as distinct.
• Stop-words Filtering: Commonly used words (e.g., "the", "an", "to") which do not add significant meaning were removed, as well as 'br', which is often found in texts from the web.
• Stemming/Lemmatization Techniques: We applied both stemming and lemmatization. Stemming truncates words to their root forms often leading to incomplete words, whereas lemmatization reduces words to their dictionary form.
• Count Vectorization: We transformed the text into vectors based on the frequency of each word appearing in the text. While simple and straightforward, this method doesn't consider the importance of word presence across multiple documents.
• TF-IDF Vectorization: This technique weights the text tokens based on their frequency, adjusted by how often they appear across all documents, thus highlighting the importance of certain words more effectively.

After testing different combinations of these methods, we found that using Lemmatization with TF-IDF Vectorization gave us the best results. It not only achieved the highest accuracy, but it also was quick.

For more detailed information and to view the step-by-step analysis, please refer to the Jupyter notebook.

C. Modeling

We selected Logistic Regression as our baseline model. It initially set a high benchmark with an accuracy of 89.35%, and it performed excellently across other metrics as well.

Model Exploration

Despite the strong performance of our baseline model, we explored additional models to ensure robustness and compare effectiveness. Below is a table summarizing the performance metrics of each model:

Model	Accuracy	Precision (Class 0)	Recall (Class 0)	Precision (Class 1)	Recall (Class 1)	F1-Score (Class 0)	F1-Score (Class 1)	AUC	Training Time (s)	Prediction Time (s)
Logistic Regression	89.35%	0.90	0.88	0.89	0.90	0.89	0.89	0.96	0.37	0.00
Gradient Boosting	80.85%	0.85	0.75	0.78	0.87	0.80	0.82	0.90	71.93	0.01
Naive Bayes	86.63%	0.86	0.88	0.88	0.85	0.87	0.86	0.94	0.02	0.00
KNN	76.92%	0.82	0.69	0.73	0.84	0.75	0.79	0.85	0.02	12.59
Decision Tree	73.12%	0.81	0.60	0.68	0.86	0.69	0.76	0.77	3.10	0.00
AdaBoost	80.01%	0.83	0.76	0.78	0.84	0.79	0.81	0.89	18.41	0.08

Model Selection

Throughout the project, Logistic Regression consistently outperformed the other models in accuracy, precision, recall, and computational efficiency. Our initial choice, proved to be the best model throughout the project, demonstrating its reliability and effectiveness in sentiment analysis.

D. Potential Business Applications and Value

The sentiment analysis model we have developed through this project offers significant potential for various business applications. Here are some ways in which businesses can utilize this model to enhance their operations and decision-making processes:

1. Customer Feedback Analysis: Companies can use this model to automatically analyze customer reviews and feedback on products and services. This can help businesses quickly identify customer sentiments, trends, and areas needing improvement, leading to more responsive customer service and product enhancements.

2. Social Media Monitoring: By using the model on social media posts and comments, businesses can understand how people feel about their brand and their competitors. This understanding helps them manage their reputation better and create marketing strategies that really speak to their audience, improving how people see their brand and increasing engagement.

3. Market Research and Stock Market Analysis: The model can analyze extensive data sets from forums, blogs, and news articles to extract public opinion about specific topics, brands, or products. Additionally, analyzing sentiments from financial news articles can help predict stock price movements. This information is invaluable for market research, helping businesses understand consumer preferences, market trends, and make informed investment decisions.

4. Enhancing User Experience: Media streaming platforms and publishers can use sentiment analysis to understand how audiences feel about different types of content. This information helps them choose and recommend content that people will enjoy more, leading to happier users and more interaction with their platform.

5. Employee Feedback Processing: Internally, companies can use this model to analyze employee feedback from surveys and forums. By knowing how employees feel, they can make the workplace better, keep morale high, and keep more employees from leaving.

Business Value

• Efficiency: Automating the analysis of textual data with our model significantly reduces the time and resources spent on manual reviews.
• Strategic Decision Making: The insights gained from sentiment analysis empower businesses to make data-driven decisions that can improve customer satisfaction, loyalty, and ultimately, profitability.
• Competitive Advantage: Using advanced analytics like sentiment analysis provides companies with a deeper understanding of the market dynamics and consumer behavior, giving them a competitive edge in their marketing and product development strategies.

By integrating sentiment analysis into their strategic operations, businesses not only optimize their existing processes but also open up new avenues for growth and innovation.

Conclusion

Data Science project showed how useful sentiment analysis can be when applied to movie reviews. We started by carefully examining the data, then moved on to improving how we handle and prepare the data for analysis.

First, we found the best way to process the data. We discovered that using lemmatization combined with TF-IDF vectorization was the most effective method. Next, we looked for the best model to predict sentiments. After testing different models, Logistic Regression proved to be the best. It was not only accurate but also worked fast, which makes it very useful for analyzing large amounts of data quickly.

We also explored how this model could help businesses. For example, it can analyze customer feedback, monitor social media, improve how content is recommended on streaming platforms, and even help in understanding employee satisfaction. These applications show that our model can really help businesses understand what people think and feel.

In simple terms, this project was not just about building a model but also about showing how it can be applied in real-world situations to help businesses make better decisions and understand their customers better. This is just the beginning, and there is a lot more we can do to make the model even better and useful in other areas too.

MLE part

Before diving into the machine learning engineering (MLE) part of this project, it's important to note that we have already defined our best data processing approach in the notebook. Therefore, for MLE components, we proceed with lemmatization and TF-IDF vectorization.

Running the Project

You have two options for running this project: locally or using Docker.

Running Locally

Follow these steps to run the project locally:

1. Create a virtual environment:

   python3 -m venv {virtual_environment_name}

2. Activate the virtual environment:
   • On Windows:

     {virtual_environment_name}\Scripts\activate

   • On macOS or Linux:

     source {virtual_environment_name}/bin/activate

3. Install Dependencies:

   pip install -r requirements.txt

4. Download Data: Training and inference data are hosted separately. To download them:

   python3 src/data_manipulation/data_upload.py

   After running the above command, a new directory named data will be created, within which the raw directory will contain two CSV files:
   • train.csv - Contains 40,000 movie reviews labeled as negative or positive.
   • inference.csv - Contains 10,000 movie reviews labeled as negative or positive.
5. Data Preprocessing: Since reviews are texts, and ML models cannot process text directly, therefore, you need to run following command:

   python3 src/data_manipulation/data_preprocess.py

   This script tokenizes the text, converts it to lowercase, removes stop words and non-alphabetical characters and lemmatizes them. For sentiment analysis, it converts them into binary labels (0 or 1). After running, the processed directory will be created in the data folder containing processed_train.csv and processed_inference.csv.
6. Data Visualization: For generating insightful visualizations, run:

   python3 src/data_manipulation/data_visualization.py

   You can see generated plots in the figures folder.
7. Model Training: To train several ML models and save the best one, run:

   python3 src/train/train.py

   This script processes processed_train.csv, splits it into training and testing sets, creates vectors with the TF-IDF vectorizer (saved in the vectorizer folder for later use to ensure consistency in feature representation between the training and inference phases. This allows the inference model to accurately interpret and process new data in the same way it handled the training data), trains several ML models, evaluates them on the test set, and saves the model with the highest accuracy in the models folder.

Note: If you want to track ML models, simply uncomment the MLflow commands in the script beforehand, and then execute mlflow ui to access the MLflow UI.

8. Inference: To run inference on unseen data, run:

   python3 src/inference/run_inference.py

This script takes processed_inference.csv, vectorizes it using the saved tf_idf_vectorizer.pkl, predicts using best_model.pkl, and saves the predictions.csv in the predictions folder along with metric.txt.

And that's it! You now have everything set up in your local environment.

Shortcut

If you prefer not to run these steps individually, simply execute:

 python3 src/main.py

This will perform all the previously described steps automatically.

Running with Docker

You can also run this project using Docker. Follow these steps:

1. Install Requirements: Install all the necessary packages with this command:

   pip install -r requirements.txt

2. Download and Preprocess Data: Download the data and preprocess it using the following commands:

   python3 src/data_manipulation/data_upload.py
   python3 src/data_manipulation/data_preprocess.py

3. Training:

• Build the Docker image for training:

  docker build -f src/train/Dockerfile -t training .

• Run the Docker container for training, mounting the outputs directory to save the results:

  docker run -v ./outputs:/app/outputs training

  After running this command, the outputs directory will be created with important files. You will have access to figures, models, and the vectorizer within this directory, allowing you to access the results of training and visualizations easily.

4. Inference:

• Build the Docker image for inference:

  docker build -f src/inference/Dockerfile -t inference .

• Run the Docker container for inference, mounting the outputs directory to collect the outputs:

  docker run -v ./outputs:/app/outputs inference

  After executing this command, the outputs directory will be updated to include a predictions subdirectory. Inside, you will find the predictions.csv file containing the inference results and a metrics.txt file documenting the best performance metric (accuracy).

This setup uses Docker to encapsulate the environment and run the project components, ensuring consistency across different machines and deployments.

Best metric

So far, based on the notebooks and scripts, the best model is logistic regression. It achieved an accuracy of 89.11% on the inference data.

Settings

The configurations for the project are managed using the settings.json file, which stores important variables that control the behavior of the project. This file includes settings for model hyperparameters, the random seed, and the test split ratio, among others. Before running the project, ensure that all paths and parameters in settings.json are correctly defined. If you wish to achieve the same results as those documented in the README, do not modify these settings.

The end 🚀