reliable_ssl_baselines

Table of Content

1. Introduction
2. Guideline
   • To prepare
   • To run experiments
   • To plot figures
   • To download checkpoints

Introduction

This repository contains the implementation of "Benchmark for Uncertainty & Robustness in Self-Supervised Learning", including a comparison of ERM, Context Prediction, Rotation Prediction, Geometric Transformation Prediction, and Jigsaw Puzzle.

Details of the model and experimental results can be found in our following arXiv paper:

@misc{bui2022benchmark,
      title={Benchmark for Uncertainty & Robustness in Self-Supervised Learning}, 
      author={Ha Manh Bui and Iliana Maifeld-Carucci},
      year={2022},
      eprint={2212.12411},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Please CITE our paper if you find it useful in your research.

Guideline

To prepare:

Install prerequisite packages:

python -m pip install -r requirements.txt

Download and unzip the datasets:

bash setup.sh

To run experiments:

Run with 10 different seeds:

for i in {1..10}; do
     taskset -c <cpu_index> python main.py --config <config_path> --exp_idx $i --gpu_idx <gpu_index>
done

where the parameters are the following:

• <cpu_index>: CPU index. E.g., <cpu_index> = "1"
• <config_path>: path stored configuration hyper-parameters. E.g., <config_path> = "algorithms/Jigsaw/configs/CIFAR10.json"
• <gpu_index>: GPU index. E.g., <gpu_index> = "0"

BERT-GPT2 experiments are provided in notebooks in "algorithms/BERT-GPT2/"

Note: Select different settings by editing in /configs/..json, logging results are stored in /results/logs/

To plot figures:

python utils/ebar_plot.py
python utils/box_plot.py
python utils/plot_density.py
python utils/plot_density_methods.py
python utils/reliability_diagram.py

To download checkpoints:

Note: All checkpoints are stored at this google drive folder.

Note: All results are recorded at this google excel.

License

This source code is released under the Apache-2.0 license, included here.