CyberDataAnalytics: lab assignment 2

Group 1: Daan van der Valk and Sandesh Manganahalli Jayaprakash

Lab assignment 2 for Cyber Data Analytics, the TU Delft course.

Installation

All included scripts should be run with Python 3. We used Python 3.6.4 to be specific, but hopefully any Python 3 version would suffice.

The following packages should be installed, which can be done using pip (pip install <package>) or Conda (conda install <package>), whatever you prefer.

• matplotlib
• scipy
• sklearn
• pydotplus
• graphviz (depending on the environment, also python-graphviz)
• imblearn
• joblib
• seaborn
• pandas

Some of these packages maybe already installed.

Highlights

Familiarization

• Map: all sensors in the dataset (generated by EPANET)
• Graph: cyclic behavior and correlation in some of the signals
• Predicting the L_T1 series using moving average (MA) is done in prediction.py
  • Graphs for a window size of 3: predictions and errors
  • Graphs for multiple window sizes: predictions and errors

ARMA

• Order estimation: arma_parameters.py resulted in arma_parameter_results.txt
  • Graphs: autocorrelation, ARMA predictions.
• Testing for all signals: arma_all_features.py resulted in arma_detection_results_all.txt

Discrete models (N-Grams)

• Discritization in percentiles: Percentiles.py resulted in discretization.svg
• trigrams.py
• fivegrams.py

PCA

• First execute: training_data03_residuals_plot.py
• Then run in the same environment: pca_all.py