DyTox

Transformers for Continual Learning with DYnamic TOken eXpansion

Welcome to DyTox, the first transformer designed explicitly for Continual Learning!

Work led by Arthur Douillard and co-authored with Alexandre Ramé, Guillaume Couairon, and Matthieu Cord.

See our erratum here.

Installation

You first need to have a working python installation with version >= 3.6.

Then create a conda env, and install the libraries in the requirements.txt: it includes pytorch and torchvision for the building blocks of our model. It also includes continuum for data loader made for continual learning, and timm.

Note that this code is heavily based on the great codebase of DeiT.

Launching an experiment

CIFAR100 dataset will be auto-downloaded, however you must download yourself ImageNet.

Each command needs three options files:

• which dataset you want to run on and in which settings (i.e. how many steps)
• Which class ordering, by default it'll be 0->C-1, but we used the class ordering proposed by DER (and which all baselines also follow)
• Which model version you want (DyTox, DyTox+, and DyTox++ (see supplementary about that last one))

To launch DyTox on CIFAR100 in the 50 steps setting on the GPUs #0 and #1:

bash train.sh 0,1 \
    --options options/data/cifar100_2-2.yaml options/data/cifar100_order1.yaml options/model/cifar_dytox.yaml \
    --name dytox \
    --data-path MY_PATH_TO_DATASET \
    --output-basedir PATH_TO_SAVE_CHECKPOINTS \
    --memory-size 1000

Folders will be auto-created with the results at logs/cifar/2-2/{DATE}/{DATE}/{DATE}-dytox.

Likewise, to launch DyTox+ and DyTox++, simply change the options. It's also the same for datasets. Note that we provided 3 different class orders (from DER's implem) for CIFAR100, and we average the results in our paper.

When you have a doubt about the options to use, just check what was defined in the yaml option files in the folder ./options.

Resuming an experiment

Some exp can be slow and you may need to resume it, like for ImageNet1000.

First locate the checkpoints folder (by default at ./checkpoints/ if you didn't specify any output-basedir) where your experiment first ran. Then run the following command (I'm taking ImageNet1000 as an example but you could have taken any models and datasets):

bash train.sh 0,1 \
    --options options/data/imagenet1000_100-100.yaml options/data/imagenet1000_order1.yaml options/model/imagenet_dytox.yaml \
    --name dytox \
    --data-path MY_PATH_TO_DATASET \
    --resume MY_PATH_TO_CKPT_FOLDER_OF_EXP \
    --start-task TASK_ID_STARTING_FROM_0_OF_WHEN_THE_EXP_HAD_STOPPED \
    --memory-size 20000

Results

ImageNet

CIFAR100

Frequenly Asked Questions

Is DyTox pretrained?

• No! It's trained from scratch for fair comparison with previous SotAs

Your encoder is made actually of ConVit blocks, can I use something else? Like a MHSA or Swin?

• Yes! I've used ConVit blocks because they trained well from scratch on small datasets like CIFAR

Can I add a new datasets?

• Yes! You can add any datasets in continual/datasets.py. They just need to be compatible with the Continuum library. But check it out, they have a lot of implemented datasets

Could I use a convolution-based backbone for the encoder instead of transformer blocks?

• Yes! You'd need to modify the DyTox module. I already provide several CNNs. Note that for best results, you may want to remove the ultimate block of the CNN and add strides so that the spatial features are big enough at the end to make enough "tokens"

Do I need to install nvidia's apex for the mix precision?

• No! DyTox uses Pytorch native mix precision

Can I run DyTox on a single GPU instead of two?

• In theory, yes. Although the performance is a bit lower. I'll try to find the root cause of this. But on two GPUs the results are perfectly reproducible.

What is this finetuning phase?

• New classes data is downsampled to the same amount of old classes data stored in the rehearsal memory. And the encoder is frozen. You can see which modules are frozen in which task in the options files.

Memory setting?

• If you use distributed memory (default), use 20/N images per class with N the number of used GPUs. Thus for 2 GPUs, it's --memory-size 1000 for CIFAR100 and ImageNet100 and --memory-size 10000 for Imagenet1000. If you use global memory (--global-memory), use 20 images per class.

Distributed memory?

• See here.

Results obtained on >=2 GPUs are slightly different from the first version of the paper?

• See here.

Citation

If you compare to this model or use this code for any means, please cite us! Thanks :)

@inproceedings{douillard2021dytox,
  title     = {DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion},
  author    = {Douillard, Arthur and Ram\'e, Alexandre and Couairon, Guillaume and Cord, Matthieu},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2022}
}