Learning Support and Trivial Prototypes for Interpretable Image Classification

Pytorch Implementation for the paper "Learning Support and Trivial Prototypes for Interpretable Image Classification" published at ICCV 2023.

In this work, we make an analogy between the prototype learning from ProtoPNet and support vector learning from SVM, and propose to learn support (i.e., hard-to-learn) prototypes, in comparison with trivial (i.e., easy-to-learn) prototypes, by forcing prototypes of different classes to locate near the classification boundary in the latent space (see figure above). In addition, we present the ST-ProtoPNet (see figure below) to exploit both support and trivial prototypes for complementary and interpretable image classification. Email: chongwangsmu@gmail.com

This repository is built mainly based on publicly available code from ProtoPNet and TesNet.

Datasets:

1. Download CUB-200-2011, Stanford Cars, and Stanford Dogs.
2. Crop images in CUB and Cars with the bounding-box information provided by the datasets: ./cropped/preprocess_data/cropimages.py.
3. Augment images in the training set in an offline mode, see details in: ./cropped/preprocess_data/img_aug.py, then put them in train_dir, test_dir, ...
4. Experiments using the raw full images in CUB and Dogs are also involved in this work, where the training images are augmented with online RandomAffine and RandomHorizontalFlip.

Train a model:

1. For cropped CUB and Cars: python ./cropped/main.py. Specifying: gpuid, backbone architecture, dataset. Note we use projection metric to compute similarity between prototypes and features in cropped CUB and Cars, as in TesNet.
2. For full CUB and Dogs: python ./full/main.py. Specifying: gpuid, backbone architecture, dataset. Note we use cosine similarity between prototypes and features for full CUB and Dogs, following Deformable ProtoPNet.

iNaturalist Pretrained ResNet-50. As in Deformable ProtoPNet and ProtoTree, we use iNaturalist-pretrained ResNet50 when training our model on full CUB-200-2011, since iNaturalist contains plants and animals and serves as a good source domain for recognising birds in CUB.

Interpretable reasoning:

1. Run local_analysis.py to find the nearest support or trivial training prototypes to a test input image, where these prototypes (as well as their source training images) will be automatically retrieved and the similarity maps of the test image with these training prototypes will be computed to realise the similarity-based interpretability.

Other experiments:

1. Measure interpretability using Intersection over Union (IoU), Content Heatmap (CH), Outside-Inside Relevance Ratio (OIRR), and Deletion AUC (DAUC).
2. Since CH, IoU, and OIRR require object annotations, we extract the binary bird segmentation masks from full CUB-200-2011.
3. Run ./full/interpretability, providing the following arguments: trained model path, test_dir, ground-truth mask path.

Citation:

@inproceedings{wang2023learning,
  title={Learning Support and Trivial Prototypes for Interpretable Image Classification},
  author={Wang, Chong and Liu, Yuyuan and Chen, Yuanhong and Liu, Fengbei and Tian, Yu and McCarthy, Davis J and Frazer, Helen and Carneiro, Gustavo},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={2062--2072},
  year={2023}
}