Unsupervised learning of action classes with continuous temporal embedding

Official implementation in python. https://arxiv.org/abs/1904.04189

If you use the code, please cite

@inproceedings{kukleva2019unsupervised,
  title={Unsupervised learning of action classes with continuous temporal embedding},
  author={Kukleva, Anna and Kuehne, Hilde and Sener, Fadime and Gall, Jurgen},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR'19)},
  year={2019}
}

To train or reproduce numbers: HowTo

Pipeline for one activity class. Figure 1 in the paper.

Proposed pipeline for unsupervised learning with unknown activity classes. Figure 2 in the paper.

Visualization of embeddings via t-SNE on the 50Salads dataset

Each frame is color coded a) with the corresponding ground truth subaction label, b) with K assigned subaction labels after clustering as the second step in Fig.1 in our main paper, c)with the predicted labels after the decoding stage. The optimization of our network is performed with respect to relative timestep of each frame. In d) we show the respective relative time label in the continuous temporal embedding assigned to each frame feature. The color bar depicts that bright blue corresponds to 0 (startof the video) and pink to 1 (end of the video).

The number of subactions K

Breakfast dataset

Activity class name	# subactions (K)
Coffe	7
Cereals	5
Tea	7
Milk	5
Juice	8
Sandwich	9
Scrambledegg	12
Friedegg	9
Salat	8
Pancake	14

YouTube Instractions dataset

Activity class name	# subactions (K)
Changing tire	11
Making cofee	10
CPR	7
Jump car	12
Repot plant	8

Qualitative results

Breakfast dataset. The order of subactions: SIL, take bowl, pour cereals, pour milk, stir cereals, SIL

Breakfast dataset. The order of subactions: SIL, take cup, add teabag, pour water, SIL

Breakfast dataset. The order of subactions: SIL, spoon powder, pour milk, stir milk, SIL

Breakfast dataset. The order of subactions: SIL, take knife, cut orange, squeeze orange, pour juice, squeeze orange, pour juice, squeeze orange, pour juice, squeeze orange, pour juice, SIL

Breakfast dataset. The order of subactions: SIL, cut bun, smear butter, put toppingOnTop, SIL

50Salads dataset. The order of subactions: start, cut, place, cut, place, cut, place, cut, place, null, null, add oil, add pepper, mix dressing, end

50Salads dataset. The order of subactions: start, cut, place, cut, place, cut, place, peel cucumber, cut, place, mix ingredients, add oil, null, add pepper, null, mix dressing, serve salad onto plate, add dressing, end