Using Gaussian Processes for Deep Neural Network Predictive Uncertainty Estimation

This projects explores a way of extracting uncertainty estimates from Deep Neural Networks (DNNs) by combining them with Gaussian Process Classifiers (GPCs), producing hybrid models. Results obtained from this project demonstrate that these hybrid models achieve high predictive accuracy on normal samples, whilst reporting high uncertainty on noisy samples. Furthermore, this project demonstrates that uncertainty estimations of these hybrid models may be used for adversarial sample detection.

Further details can be found in the corresponding report.

Setup

Prerequisites:

Ensure you have the following packages installed (these can all be installed with pip3):

• Keras
• Cleverhans
• GPFlow
• pyYAML
• SciPy

MNIST Model

This project requires a trained MNIST classification model, which can be generated using the mnist_cnn.py file, found here.

Usage

• Create MNIST model, as specified in the previous section
• Specify it's path in the config.yml file:

model_name: "path-to-generated-model"

• Optionally, download the NMNIST dataset (available here), and specify it's path in the config.yml file:

nmnist_path: "path-to-nmnist-dataset"

• Navigate into the repository directory
• Run main.py:

python3 ./main.py

Files/Directories

• gpc_models: directory includes files for training/running a single Gaussian Process Classifier (single_GP.py), or an ensemble of such classifiers (ensemble_GP.py)

• sample_generator: includes the adv_sample_generator.py file, which can be used for generating adversarial samples using the CleverHans toolbox

• utils: includes the utils.py file, which is used for loading and parsing the MNIST and NMNIST datasets

• config.yml: configuration file storing path names

• main.py: file used for loading and running the GPC-DNN hybrid model

• report.pdf: a corresponding project report, discussing the relevant findings