notebooks/AV_FinalContest_AgeEstimation_Training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AV_FinalContest_AgeEstimation_Training.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-vWPVwRLsv5j"
      },
      "source": [
        "## **Introduction**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ihcze5gzs04P"
      },
      "source": [
        "This notebook has been created for the final contest of Artificial Vision subject at University of Salerno.The aim of this project is to design a DCNN (as regressor or classifier) for age estimation on [VggFace2 dataset](https://github.com/ox-vgg/vgg_face2) labeled with ages by [MiviaLab](https://mivia.unisa.it/).\r\n",
        "\r\n",
        "<br/>\r\n",
        "\r\n",
        "We decided to build a classifier able to recognize 101 classes (ages from 0 to 100), in particular we choose the [Resnet50 model](https://github.com/WeidiXie/Keras-VGGFace2-ResNet50).\r\n",
        "\r\n",
        "In this notebook, we show our **training procedure**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fapW2uczvf8D"
      },
      "source": [
        "## **Initialization**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fGQddyujvoat"
      },
      "source": [
        "First of all, we have to mount the Drive and to go in the folder where all operations has to be done because it contains all the needed files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yUuqazWWsW7Y",
        "outputId": "4b1a7822-416a-4dce-c808-251aee4ff6c8"
      },
      "source": [
        "from google.colab import drive\r\n",
        "import os\r\n",
        "\r\n",
        "drive.mount('/content/drive')\r\n",
        "os.chdir('/content/drive/Shareddrives/ArtificialVision/FinalContest2020')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5LXmGjRywhij"
      },
      "source": [
        "Check if we are using a GPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sIEp-NnvwfPS"
      },
      "source": [
        "%tensorflow_version 2.x\r\n",
        "import tensorflow as tf\r\n",
        "\r\n",
        "device_name = tf.test.gpu_device_name()\r\n",
        "if device_name != '/device:GPU:0':\r\n",
        "  raise SystemError('GPU device not found')\r\n",
        "print('Found GPU at: {}'.format(device_name))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eTxjZMfQELsw"
      },
      "source": [
        "Cloning the Github repository of the GenderRecognitionFramework provided by MiviaLab, containing some code used later in this notebook"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LjR4uEwUEhXc",
        "outputId": "46d55029-3011-45f3-97ed-7f1604973cef"
      },
      "source": [
        "!git clone https://github.com/MiviaLab/GenderRecognitionFramework"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'GenderRecognitionFramework'...\n",
            "remote: Enumerating objects: 219, done.\u001b[K\n",
            "remote: Counting objects: 100% (219/219), done.\u001b[K\n",
            "remote: Compressing objects: 100% (174/174), done.\u001b[K\n",
            "remote: Total 219 (delta 38), reused 202 (delta 30), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (219/219), 9.01 MiB | 16.96 MiB/s, done.\n",
            "Resolving deltas: 100% (38/38), done.\n",
            "Checking out files: 100% (165/165), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BGVPfBvhwxM6"
      },
      "source": [
        "## **Create Resnet50 model**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5r2SGum5wMr"
      },
      "source": [
        "Define some useful parameters, according to CNN model and problem definition respectively"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yEVnsAU7w5I1"
      },
      "source": [
        "TARGET_SHAPE = (224, 224, 3)\r\n",
        "NUM_CLASSES = 101"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uNmF1sTP6DVx"
      },
      "source": [
        "Cloning Github repository containing the code for building CNN model architecture chosen"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XYBSnTcnw9xU",
        "outputId": "60328a99-f52b-444c-b198-92b4dc7e90d3"
      },
      "source": [
        "!git clone https://github.com/WeidiXie/Keras-VGGFace2-ResNet50"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'Keras-VGGFace2-ResNet50'...\n",
            "remote: Enumerating objects: 12, done.\u001b[K\n",
            "remote: Counting objects: 100% (12/12), done.\u001b[K\n",
            "remote: Compressing objects: 100% (12/12), done.\u001b[K\n",
            "remote: Total 158 (delta 6), reused 0 (delta 0), pack-reused 146\u001b[K\n",
            "Receiving objects: 100% (158/158), 36.73 MiB | 15.96 MiB/s, done.\n",
            "Resolving deltas: 100% (43/43), done.\n",
            "Checking out files: 100% (79/79), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1ggSLDC6Qzi"
      },
      "source": [
        "Build model architecture, using functions provided by Github repository and then adding a final dense layer of 101 neurons for adapting the pre-trained net to solve our problem.\r\n",
        "\r\n",
        "**N.B.** : *Vggface2_ResNet50()* function has been modified deleting some LOCs referring to model compiling that, for convenience, has been inserted later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O9TNcFJzxatR"
      },
      "source": [
        "import sys\r\n",
        "sys.path.append(\"./Keras-VGGFace2-ResNet50/src/\")\r\n",
        "from resnet import resnet50_backend\r\n",
        "from model import Vggface2_ResNet50\r\n",
        "\r\n",
        "from keras.models import Model\r\n",
        "from keras.layers import Dense\r\n",
        "\r\n",
        "# build the model, loading pre-trained weights on ImageNet or the ones passed as parameter\r\n",
        "# and adding a final dense layer of 101 neurons with softmax activation function\r\n",
        "def get_model(num_classes, weights_path=None):\r\n",
        "  model_resnet = Vggface2_ResNet50()\r\n",
        "  if weights_path != None:\r\n",
        "    model_resnet.load_weights(weights_path)\r\n",
        "  else:\r\n",
        "    model_resnet.load_weights('weights.h5')\r\n",
        "  \r\n",
        "  model = Dense(num_classes, activation='softmax')(model_resnet.layers[-2].output)\r\n",
        "  model = Model(inputs=model_resnet.input, outputs=model)\r\n",
        "\r\n",
        "  # first 18 epochs of training\r\n",
        "  '''\r\n",
        "  for layer in model.layers[:-11]:\r\n",
        "    layer.trainable = False\r\n",
        "  '''\r\n",
        "  \r\n",
        "  # following 7 epochs of training\r\n",
        "  for layer in model.layers:\r\n",
        "    layer.trainable = True\r\n",
        "\r\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CV6wrmMlz_A9",
        "outputId": "9a96d81c-d500-4d09-a0c5-dd9b82a188ad"
      },
      "source": [
        "model = get_model(NUM_CLASSES)\r\n",
        "print(model.summary())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "base_input (InputLayer)         [(None, 224, 224, 3) 0                                            \n",
            "__________________________________________________________________________________________________\n",
            "conv1/7x7_s2 (Conv2D)           (None, 112, 112, 64) 9408        base_input[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "conv1/7x7_s2/bn (BatchNormaliza (None, 112, 112, 64) 256         conv1/7x7_s2[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "activation (Activation)         (None, 112, 112, 64) 0           conv1/7x7_s2/bn[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling2d (MaxPooling2D)    (None, 55, 55, 64)   0           activation[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_1x1_reduce (Conv2D)     (None, 55, 55, 64)   4096        max_pooling2d[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_1x1_reduce/bn (BatchNor (None, 55, 55, 64)   256         conv2_1_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_1 (Activation)       (None, 55, 55, 64)   0           conv2_1_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_3x3 (Conv2D)            (None, 55, 55, 64)   36864       activation_1[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_3x3/bn (BatchNormalizat (None, 55, 55, 64)   256         conv2_1_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_2 (Activation)       (None, 55, 55, 64)   0           conv2_1_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_1x1_increase (Conv2D)   (None, 55, 55, 256)  16384       activation_2[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_1x1_proj (Conv2D)       (None, 55, 55, 256)  16384       max_pooling2d[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_1x1_increase/bn (BatchN (None, 55, 55, 256)  1024        conv2_1_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "conv2_1_1x1_proj/bn (BatchNorma (None, 55, 55, 256)  1024        conv2_1_1x1_proj[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "add (Add)                       (None, 55, 55, 256)  0           conv2_1_1x1_increase/bn[0][0]    \n",
            "                                                                 conv2_1_1x1_proj/bn[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "activation_3 (Activation)       (None, 55, 55, 256)  0           add[0][0]                        \n",
            "__________________________________________________________________________________________________\n",
            "conv2_2_1x1_reduce (Conv2D)     (None, 55, 55, 64)   16384       activation_3[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_2_1x1_reduce/bn (BatchNor (None, 55, 55, 64)   256         conv2_2_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_4 (Activation)       (None, 55, 55, 64)   0           conv2_2_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv2_2_3x3 (Conv2D)            (None, 55, 55, 64)   36864       activation_4[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_2_3x3/bn (BatchNormalizat (None, 55, 55, 64)   256         conv2_2_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_5 (Activation)       (None, 55, 55, 64)   0           conv2_2_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv2_2_1x1_increase (Conv2D)   (None, 55, 55, 256)  16384       activation_5[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_2_1x1_increase/bn (BatchN (None, 55, 55, 256)  1024        conv2_2_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_1 (Add)                     (None, 55, 55, 256)  0           conv2_2_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_3[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "activation_6 (Activation)       (None, 55, 55, 256)  0           add_1[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv2_3_1x1_reduce (Conv2D)     (None, 55, 55, 64)   16384       activation_6[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_3_1x1_reduce/bn (BatchNor (None, 55, 55, 64)   256         conv2_3_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_7 (Activation)       (None, 55, 55, 64)   0           conv2_3_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv2_3_3x3 (Conv2D)            (None, 55, 55, 64)   36864       activation_7[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_3_3x3/bn (BatchNormalizat (None, 55, 55, 64)   256         conv2_3_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_8 (Activation)       (None, 55, 55, 64)   0           conv2_3_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv2_3_1x1_increase (Conv2D)   (None, 55, 55, 256)  16384       activation_8[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv2_3_1x1_increase/bn (BatchN (None, 55, 55, 256)  1024        conv2_3_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_2 (Add)                     (None, 55, 55, 256)  0           conv2_3_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_6[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "activation_9 (Activation)       (None, 55, 55, 256)  0           add_2[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_1x1_reduce (Conv2D)     (None, 28, 28, 128)  32768       activation_9[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_1_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_10 (Activation)      (None, 28, 28, 128)  0           conv3_1_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_10[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_1_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_11 (Activation)      (None, 28, 28, 128)  0           conv3_1_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_11[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_1x1_proj (Conv2D)       (None, 28, 28, 512)  131072      activation_9[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_1_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "conv3_1_1x1_proj/bn (BatchNorma (None, 28, 28, 512)  2048        conv3_1_1x1_proj[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "add_3 (Add)                     (None, 28, 28, 512)  0           conv3_1_1x1_increase/bn[0][0]    \n",
            "                                                                 conv3_1_1x1_proj/bn[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "activation_12 (Activation)      (None, 28, 28, 512)  0           add_3[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_2_1x1_reduce (Conv2D)     (None, 28, 28, 128)  65536       activation_12[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_2_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_2_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_13 (Activation)      (None, 28, 28, 128)  0           conv3_2_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_2_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_13[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_2_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_2_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_14 (Activation)      (None, 28, 28, 128)  0           conv3_2_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv3_2_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_14[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_2_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_2_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_4 (Add)                     (None, 28, 28, 512)  0           conv3_2_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_12[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_15 (Activation)      (None, 28, 28, 512)  0           add_4[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_3_1x1_reduce (Conv2D)     (None, 28, 28, 128)  65536       activation_15[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_3_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_3_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_16 (Activation)      (None, 28, 28, 128)  0           conv3_3_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_3_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_16[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_3_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_3_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_17 (Activation)      (None, 28, 28, 128)  0           conv3_3_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv3_3_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_17[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_3_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_3_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_5 (Add)                     (None, 28, 28, 512)  0           conv3_3_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_15[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_18 (Activation)      (None, 28, 28, 512)  0           add_5[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_4_1x1_reduce (Conv2D)     (None, 28, 28, 128)  65536       activation_18[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_4_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_4_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_19 (Activation)      (None, 28, 28, 128)  0           conv3_4_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv3_4_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_19[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_4_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_4_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_20 (Activation)      (None, 28, 28, 128)  0           conv3_4_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv3_4_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_20[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv3_4_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_4_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_6 (Add)                     (None, 28, 28, 512)  0           conv3_4_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_18[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_21 (Activation)      (None, 28, 28, 512)  0           add_6[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_1x1_reduce (Conv2D)     (None, 14, 14, 256)  131072      activation_21[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_1_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_22 (Activation)      (None, 14, 14, 256)  0           conv4_1_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_22[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_1_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_23 (Activation)      (None, 14, 14, 256)  0           conv4_1_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_23[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_1x1_proj (Conv2D)       (None, 14, 14, 1024) 524288      activation_21[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_1_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "conv4_1_1x1_proj/bn (BatchNorma (None, 14, 14, 1024) 4096        conv4_1_1x1_proj[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "add_7 (Add)                     (None, 14, 14, 1024) 0           conv4_1_1x1_increase/bn[0][0]    \n",
            "                                                                 conv4_1_1x1_proj/bn[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "activation_24 (Activation)      (None, 14, 14, 1024) 0           add_7[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_2_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_24[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_2_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_2_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_25 (Activation)      (None, 14, 14, 256)  0           conv4_2_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_2_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_25[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_2_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_2_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_26 (Activation)      (None, 14, 14, 256)  0           conv4_2_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv4_2_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_26[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_2_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_2_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_8 (Add)                     (None, 14, 14, 1024) 0           conv4_2_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_24[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_27 (Activation)      (None, 14, 14, 1024) 0           add_8[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_3_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_27[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_3_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_3_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_28 (Activation)      (None, 14, 14, 256)  0           conv4_3_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_3_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_28[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_3_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_3_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_29 (Activation)      (None, 14, 14, 256)  0           conv4_3_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv4_3_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_29[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_3_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_3_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_9 (Add)                     (None, 14, 14, 1024) 0           conv4_3_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_27[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_30 (Activation)      (None, 14, 14, 1024) 0           add_9[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_4_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_30[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_4_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_4_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_31 (Activation)      (None, 14, 14, 256)  0           conv4_4_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_4_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_31[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_4_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_4_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_32 (Activation)      (None, 14, 14, 256)  0           conv4_4_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv4_4_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_32[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_4_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_4_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_10 (Add)                    (None, 14, 14, 1024) 0           conv4_4_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_30[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_33 (Activation)      (None, 14, 14, 1024) 0           add_10[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv4_5_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_33[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_5_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_5_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_34 (Activation)      (None, 14, 14, 256)  0           conv4_5_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_5_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_34[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_5_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_5_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_35 (Activation)      (None, 14, 14, 256)  0           conv4_5_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv4_5_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_35[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_5_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_5_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_11 (Add)                    (None, 14, 14, 1024) 0           conv4_5_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_33[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_36 (Activation)      (None, 14, 14, 1024) 0           add_11[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv4_6_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_36[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_6_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_6_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_37 (Activation)      (None, 14, 14, 256)  0           conv4_6_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv4_6_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_37[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_6_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_6_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_38 (Activation)      (None, 14, 14, 256)  0           conv4_6_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv4_6_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_38[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv4_6_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_6_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_12 (Add)                    (None, 14, 14, 1024) 0           conv4_6_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_36[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_39 (Activation)      (None, 14, 14, 1024) 0           add_12[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_1x1_reduce (Conv2D)     (None, 7, 7, 512)    524288      activation_39[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_1x1_reduce/bn (BatchNor (None, 7, 7, 512)    2048        conv5_1_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_40 (Activation)      (None, 7, 7, 512)    0           conv5_1_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_3x3 (Conv2D)            (None, 7, 7, 512)    2359296     activation_40[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_3x3/bn (BatchNormalizat (None, 7, 7, 512)    2048        conv5_1_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_41 (Activation)      (None, 7, 7, 512)    0           conv5_1_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_1x1_increase (Conv2D)   (None, 7, 7, 2048)   1048576     activation_41[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_1x1_proj (Conv2D)       (None, 7, 7, 2048)   2097152     activation_39[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_1x1_increase/bn (BatchN (None, 7, 7, 2048)   8192        conv5_1_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "conv5_1_1x1_proj/bn (BatchNorma (None, 7, 7, 2048)   8192        conv5_1_1x1_proj[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "add_13 (Add)                    (None, 7, 7, 2048)   0           conv5_1_1x1_increase/bn[0][0]    \n",
            "                                                                 conv5_1_1x1_proj/bn[0][0]        \n",
            "__________________________________________________________________________________________________\n",
            "activation_42 (Activation)      (None, 7, 7, 2048)   0           add_13[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv5_2_1x1_reduce (Conv2D)     (None, 7, 7, 512)    1048576     activation_42[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_2_1x1_reduce/bn (BatchNor (None, 7, 7, 512)    2048        conv5_2_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_43 (Activation)      (None, 7, 7, 512)    0           conv5_2_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv5_2_3x3 (Conv2D)            (None, 7, 7, 512)    2359296     activation_43[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_2_3x3/bn (BatchNormalizat (None, 7, 7, 512)    2048        conv5_2_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_44 (Activation)      (None, 7, 7, 512)    0           conv5_2_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv5_2_1x1_increase (Conv2D)   (None, 7, 7, 2048)   1048576     activation_44[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_2_1x1_increase/bn (BatchN (None, 7, 7, 2048)   8192        conv5_2_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_14 (Add)                    (None, 7, 7, 2048)   0           conv5_2_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_42[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_45 (Activation)      (None, 7, 7, 2048)   0           add_14[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "conv5_3_1x1_reduce (Conv2D)     (None, 7, 7, 512)    1048576     activation_45[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_3_1x1_reduce/bn (BatchNor (None, 7, 7, 512)    2048        conv5_3_1x1_reduce[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "activation_46 (Activation)      (None, 7, 7, 512)    0           conv5_3_1x1_reduce/bn[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "conv5_3_3x3 (Conv2D)            (None, 7, 7, 512)    2359296     activation_46[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_3_3x3/bn (BatchNormalizat (None, 7, 7, 512)    2048        conv5_3_3x3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "activation_47 (Activation)      (None, 7, 7, 512)    0           conv5_3_3x3/bn[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "conv5_3_1x1_increase (Conv2D)   (None, 7, 7, 2048)   1048576     activation_47[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "conv5_3_1x1_increase/bn (BatchN (None, 7, 7, 2048)   8192        conv5_3_1x1_increase[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "add_15 (Add)                    (None, 7, 7, 2048)   0           conv5_3_1x1_increase/bn[0][0]    \n",
            "                                                                 activation_45[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "activation_48 (Activation)      (None, 7, 7, 2048)   0           add_15[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "avg_pool (AveragePooling2D)     (None, 1, 1, 2048)   0           activation_48[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "flatten (Flatten)               (None, 2048)         0           avg_pool[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "dim_proj (Dense)                (None, 512)          1049088     flatten[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, 101)          51813       dim_proj[0][0]                   \n",
            "==================================================================================================\n",
            "Total params: 24,662,053\n",
            "Trainable params: 24,608,933\n",
            "Non-trainable params: 53,120\n",
            "__________________________________________________________________________________________________\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oYB25mRm821q"
      },
      "source": [
        "## **Parameters, Callbacks and Model Compiling with them**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YLsKkmj4ALuD"
      },
      "source": [
        "### Definition of useful parameters for training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bV5Kknh88550"
      },
      "source": [
        "#@title Parameters used in model compile\r\n",
        "\r\n",
        "n_epochs = 25 #@param {type:\"integer\"}\r\n",
        "batch_size = 128 #@param {type:\"integer\"}\r\n",
        "initial_learning_rate = 0.005 #@param {type:\"number\"}\r\n",
        "learning_rate_decay_factor = 0.2 #@param {type:\"number\"}\r\n",
        "learning_rate_decay_epochs = 20 #@param {type:\"number\"}\r\n",
        "momentum = 0.9 #@param {type:\"number\"}\r\n",
        "patience =  5 #@param {type:\"integer\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LzZ9irtY9sU3"
      },
      "source": [
        "#@title Parameters for logging directory\r\n",
        "\r\n",
        "log_dir = \"./logs/resnet50/\" #@param {type:\"string\"}\r\n",
        "mode = \"training\" #@param {type:\"string\"}\r\n",
        "\r\n",
        "if not os.path.isdir(log_dir): os.mkdir(log_dir)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9TyWS8Fc-aaL"
      },
      "source": [
        "Creation of the directory used for storing model checkpints"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZIOXoz5A-W8y",
        "outputId": "c368cadd-bfa4-485c-db17-5ac793fda125"
      },
      "source": [
        "from glob import glob\r\n",
        "import re\r\n",
        "\r\n",
        "dirnm = mode+\"_logs/\"\r\n",
        "dirnm = os.path.join(log_dir, dirnm) #./logs/<net>/inference-training/\r\n",
        "print(\"Log dir: {}\".format(dirnm))\r\n",
        "if not os.path.isdir(dirnm): os.mkdir(dirnm)\r\n",
        "\r\n",
        "chk_dir = dirnm + \"weights/\" #./logs/<net>/inference-training/weights/\r\n",
        "print(\"Checkpoint dir: {}\".format(chk_dir))\r\n",
        "if not os.path.isdir(chk_dir): os.mkdir(chk_dir)\r\n",
        "\r\n",
        "filepath = os.path.join(chk_dir, \"checkpoint.{epoch:02d}.h5\")\r\n",
        "ep_re = re.compile('checkpoint.([0-9]+).h5')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Log dir: ./logs/resnet50/training_logs/\n",
            "Checkpoint dir: ./logs/resnet50/training_logs/weights/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_kUjzl7_TKJ"
      },
      "source": [
        "This code allows to resume a previous started training from the last saved checkpoint done"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FrqV_BnR_d59"
      },
      "source": [
        "resume = True #@param {type:\"boolean\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ux3YTiEh_2-4",
        "outputId": "694ea2ed-1644-407d-86dd-b7a1e3d625e0"
      },
      "source": [
        "if resume:\r\n",
        "  pattern = filepath.replace('{epoch:02d}', '*')\r\n",
        "  epochs = glob(pattern)\r\n",
        "  epochs = [int(x[-6:-3].replace('.', '')) for x in epochs]\r\n",
        "  initial_epoch = max(epochs)\r\n",
        "  print('Resuming from epoch %d...' % initial_epoch)\r\n",
        "  model.load_weights(filepath.format(epoch=initial_epoch))\r\n",
        "else:\r\n",
        "  print(\"Training from scratch\")\r\n",
        "  initial_epoch = 0"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Resuming from epoch 25...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D3dT-l4BAI7O"
      },
      "source": [
        "### Callbacks definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wlHiv8hQAxjs"
      },
      "source": [
        "As a metric for evaluating the model, we use the MAE but the Keras implementation is not good for our purpose because we have adopted an one-hot encoding for labels but  we have as CNN output a vector of probabilities (as softmax activation function consequence), not 1 in the predicted class position and 0 otherwise.\r\n",
        "\r\n",
        "For this reason we implement a custom MAE, which relies on Keras one."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vYaWQEBzB_Nt"
      },
      "source": [
        "import keras.backend as K\r\n",
        "from tensorflow.keras.metrics import mean_absolute_error\r\n",
        "\r\n",
        "@tf.function\r\n",
        "def mae(y_true, y_pred):\r\n",
        "  y_true = K.cast(y_true, y_pred.dtype)\r\n",
        "\r\n",
        "  # as input we have a batch of arrays of 101 elements,\r\n",
        "  # so for each array we extract the max probability as index for the predicted class\r\n",
        "  ages_true = tf.map_fn(lambda true: K.argmax(true), y_true, dtype=tf.int64)\r\n",
        "  ages_pred = tf.map_fn(lambda pred: K.argmax(pred), y_pred, dtype=tf.int64)\r\n",
        "\r\n",
        "  return mean_absolute_error(ages_true, ages_pred)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1llY36n3DNoq"
      },
      "source": [
        "This function allows to reduce the initial learning rate by a given factor after some epochs.\r\n",
        "\r\n",
        "\r\n",
        "**N.B.** It is taken from /GenderRecognitionFramework/training/train.py file but has been copied here for solving technical problems. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XD7JoZOYFL9r"
      },
      "source": [
        "def step_decay_schedule(initial_lr, decay_factor, step_size):\r\n",
        "    def schedule(epoch):\r\n",
        "        return initial_lr * (decay_factor ** np.floor(epoch / step_size))\r\n",
        "\r\n",
        "    return tf.keras.callbacks.LearningRateScheduler(schedule, verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1NfWhR_GlGQ"
      },
      "source": [
        "Compile previous created model with:\r\n",
        "\r\n",
        "* *parameters*:\r\n",
        "  * optimizer -> SGD with momentum\r\n",
        "  * loss -> Categorical Cross Entropy\r\n",
        "  * metrics -> Categorical Accuracy, MAE\r\n",
        "\r\n",
        "* *callbacks*:\r\n",
        "  * learning rate scheduling\r\n",
        "  * model checkpoint saving\r\n",
        "  * early stopping\r\n",
        "\r\n",
        "\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kO6Sz-YnGEbM"
      },
      "source": [
        "from keras.optimizers import SGD\r\n",
        "from keras.callbacks import TensorBoard, ModelCheckpoint, EarlyStopping\r\n",
        "%tensorflow_version 2.x\r\n",
        "import tensorflow as tf\r\n",
        "from tensorflow import keras\r\n",
        "\r\n",
        "# Compilation parameters    \r\n",
        "optimizer = tf.keras.optimizers.SGD(momentum=momentum)\r\n",
        "loss = tf.keras.losses.categorical_crossentropy\r\n",
        "metrics = [keras.metrics.categorical_accuracy, mae]\r\n",
        "\r\n",
        "# Callbacks\r\n",
        "lr_sched = step_decay_schedule(initial_lr=initial_learning_rate, decay_factor=learning_rate_decay_factor, step_size=learning_rate_decay_epochs)\r\n",
        "checkpoint = tf.keras.callbacks.ModelCheckpoint(\r\n",
        "    filepath, save_best_only=True, monitor=\"val_categorical_accuracy\", mode='max'\r\n",
        ")\r\n",
        "tbCallBack = tf.keras.callbacks.TensorBoard(log_dir=log_dir, write_graph=True, write_images=True)\r\n",
        "early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', mode=\"min\", patience=patience)\r\n",
        "callbacks_list = [lr_sched, checkpoint, tbCallBack, early_stopping]\r\n",
        "\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HlOEuoLzGSqF"
      },
      "source": [
        "model.compile(\r\n",
        "  loss=loss, \r\n",
        "  optimizer=optimizer, \r\n",
        "  metrics = metrics\r\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "97UcoRyMH6_7"
      },
      "source": [
        "## **Utilities definition**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jP7jRq5a3t1"
      },
      "source": [
        "This function allows to resize the image passed as input to a desired size; if image's size is less than target shape, it adds a black padding around the image, otherwise it resizes the images with an antialias filter before"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_aDysyE8H-YJ"
      },
      "source": [
        "%tensorflow_version 2.x\r\n",
        "import tensorflow as tf\r\n",
        "\r\n",
        "def custom_resize(image, h, w, target_shape):\r\n",
        "  t_h, t_w = target_shape[0], target_shape[1]\r\n",
        "  if h < t_h and w < t_w:\r\n",
        "    image = tf.image.resize_with_crop_or_pad(image, t_h, t_w)\r\n",
        "  else:\r\n",
        "    image = tf.image.resize(image, (t_h, t_w), antialias=True)\r\n",
        "  return image"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2V_SsDHbgzu"
      },
      "source": [
        "This function preprocess the image passed as input, applying a normalization consisting in the subtraction of the mean of 3 color channel over all the dataset. It's taken from /GenderRecognitionFramework/training/dataset_tools.py and reported here for convenience."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lHsOlMxHIIQP"
      },
      "source": [
        "import numpy as np\r\n",
        "import sys\r\n",
        "sys.path.append(\"./GenderRecognitionFramework/training\")\r\n",
        "from dataset_tools import mean_std_normalize\r\n",
        "\r\n",
        "def vggface2_preprocessing(img):\r\n",
        "  ds_means = np.array([131.0912, 103.8827, 91.4953]) # RGB\r\n",
        "  ds_stds = None\r\n",
        "  img = mean_std_normalize(img, ds_means, ds_stds)\r\n",
        "  if (len(img.shape)<3 or img.shape[2]<3):\r\n",
        "      img = np.repeat(np.squeeze(img)[:,:,None], 3, axis=2)\r\n",
        "  return img"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRBHIOSjc0zL"
      },
      "source": [
        "This is an alternative augmentation, tried for some epochs during the training experiments done but then discarded in favour of the other implementation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ab2erbTZDSE6"
      },
      "source": [
        "def flip(img):\r\n",
        "    img=np.fliplr(img)\r\n",
        "    return img\r\n",
        "\r\n",
        "def monochrome(x):\r\n",
        "    x = cv2.cvtColor(x, cv2.COLOR_RGB2GRAY)\r\n",
        "    if len(x.shape)==2:\r\n",
        "        x = x[:,:,np.newaxis]\r\n",
        "    x = np.repeat(x, 3, axis=2)\r\n",
        "    return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8XwzL8_fOdp"
      },
      "source": [
        "## **Dataset management**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nuMRfnKkiD44"
      },
      "source": [
        "This code block allows to create a dataset from the TFRecord file uploaded on Drive, distinguishing between test and validation/training TFRecord because they have different record format"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EbIg9ksmfUYP"
      },
      "source": [
        "%tensorflow_version 2.x\r\n",
        "import tensorflow as tf\r\n",
        "from functools import partial\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "def read_tfrecord_test(example):\r\n",
        "    tfrecord_format = (\r\n",
        "        {\r\n",
        "          'path': tf.io.FixedLenFeature([], tf.string),\r\n",
        "          'image_raw': tf.io.FixedLenFeature([], tf.string),\r\n",
        "        }\r\n",
        "    )    \r\n",
        "    return tf.io.parse_single_example(example, tfrecord_format)\r\n",
        "    \r\n",
        "def read_tfrecord(example):\r\n",
        "    tfrecord_format = (\r\n",
        "        {\r\n",
        "          'path': tf.io.FixedLenFeature([], tf.string),\r\n",
        "          'height': tf.io.FixedLenFeature([], tf.int64),\r\n",
        "          'width': tf.io.FixedLenFeature([], tf.int64),\r\n",
        "          'label': tf.io.FixedLenFeature([], tf.int64),\r\n",
        "          'image_raw': tf.io.FixedLenFeature([], tf.string),\r\n",
        "        }\r\n",
        "    )\r\n",
        "    return tf.io.parse_single_example(example, tfrecord_format)\r\n",
        "\r\n",
        "def load_dataset(filenames, test):\r\n",
        "    ignore_order = tf.data.Options()\r\n",
        "    ignore_order.experimental_deterministic = False\r\n",
        "    dataset = tf.data.TFRecordDataset(filenames) # create dataset from path passed as input\r\n",
        "    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order\r\n",
        "    \r\n",
        "    # read dataset records according to its type (test or not)\r\n",
        "    if not test:\r\n",
        "      dataset = dataset.map(partial(read_tfrecord))\r\n",
        "    else:\r\n",
        "      dataset = dataset.map(partial(read_tfrecord_test))\r\n",
        "    return dataset\r\n",
        "\r\n",
        "def get_dataset(filenames, dataset_dim, test=False):\r\n",
        "    dataset = load_dataset(filenames, test) \r\n",
        "    if not test: #shuffle elements at each epoch\r\n",
        "      dataset = dataset.shuffle(dataset_dim//256, reshuffle_each_iteration=True).repeat()\r\n",
        "    #This allows later elements to be prepared while the current element is being processed.\r\n",
        "    dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE) \r\n",
        "    if not test: # set batch size\r\n",
        "      dataset = dataset.batch(batch_size)\r\n",
        "    return dataset\r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYSG-o8akxWO"
      },
      "source": [
        "Creation of training and validation dataset from TFRecord"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tjp0APXpkBtm"
      },
      "source": [
        "TRAINING_FILENAMES = \"tfrecords/training_set_cropped.record\"\r\n",
        "VALID_FILENAMES = \"tfrecords/validation_set_cropped.record\" \r\n",
        "\r\n",
        "# parameters due to our subset of samples\r\n",
        "tot_train_sample = 790487\r\n",
        "tot_valid_sample = 344795\r\n",
        "\r\n",
        "train_dataset = get_dataset(TRAINING_FILENAMES, tot_train_sample)\r\n",
        "valid_dataset = get_dataset(VALID_FILENAMES, tot_valid_sample)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F7HoDTS6ij_m"
      },
      "source": [
        "## Data Iterator with Augmentation and Pre-Processing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QL3uCbaVi36b"
      },
      "source": [
        "This code block creates an iterator over the prevously created dataset, applying:\r\n",
        "* *for each image*, augmentation, normalization and resize\r\n",
        "* *for each label*, conversion to one-hot encoding\r\n",
        "\r\n",
        "**N.B.** The function *random_monochrome()* taken from Mivia framework has a modification at line 180 for adapting it to our dataset "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9GHiJ7p2ik3V"
      },
      "source": [
        "# create data generator\r\n",
        "import numpy as np\r\n",
        "from keras.utils import Sequence\r\n",
        "import random\r\n",
        "\r\n",
        "import sys\r\n",
        "sys.path.append(\"./AgeRecognitionFramework/training\")\r\n",
        "from dataset_tools import random_flip, random_monochrome, random_brightness_contrast\r\n",
        "\r\n",
        "class Generator(Sequence):\r\n",
        "    def __init__(self, batch_size, elements, dataset, validation):\r\n",
        "      self.batch_size = batch_size\r\n",
        "      self.dataset_iterator = iter(dataset)\r\n",
        "      self.elements = elements\r\n",
        "      self.validation = validation\r\n",
        "\r\n",
        "    def __len__(self):\r\n",
        "      return int(np.ceil(self.elements// (self.batch_size)))\r\n",
        "\r\n",
        "    def data_augmentation(self, image): #potentially all augmentations\r\n",
        "        img = random_brightness_contrast(image)\r\n",
        "        img = random_monochrome(img)\r\n",
        "        return random_flip(img)\r\n",
        "\r\n",
        "    def custom_augmentation(self, image): #only a kind of augmentation but surely\r\n",
        "        aug = random.randint(1,3)\r\n",
        "        if aug == 1:\r\n",
        "          img = random_brightness_contrast(image)\r\n",
        "        elif aug == 2:\r\n",
        "          img = monochrome(image)\r\n",
        "        else:\r\n",
        "          img = flip(image)\r\n",
        "        return img\r\n",
        "\r\n",
        "    def decode_image(self, image): #decode bytes to image\r\n",
        "        image = tf.image.decode_jpeg(image, channels=3)\r\n",
        "        return image\r\n",
        "\r\n",
        "    def __getitem__(self, idx):\r\n",
        "        custom = False #True=augmentation with only one kind randomly chosen\r\n",
        "\r\n",
        "        parsing_dict = self.dataset_iterator.get_next() #take a batch of TFRecord\r\n",
        "        \r\n",
        "        #read each image-label of the batch\r\n",
        "        images = []\r\n",
        "        labels = []\r\n",
        "        for i in range(self.batch_size): \r\n",
        "          images.append((self.decode_image(parsing_dict[\"image_raw\"][i])).numpy())\r\n",
        "          labels.append(parsing_dict[\"label\"][i])\r\n",
        "\r\n",
        "        # process images&labels\r\n",
        "        processed_images = []\r\n",
        "        processed_labels = []        \r\n",
        "        \r\n",
        "        # process images\r\n",
        "        for img in images:\r\n",
        "          if not self.validation: #if it's a validation generator, not apply augmentation\r\n",
        "            to_aug = random.randint(0,1) #random decision between aug e no_aug\r\n",
        "            if to_aug:\r\n",
        "              if custom:\r\n",
        "                img = self.custom_augmentation(img)\r\n",
        "              else:\r\n",
        "                img = self.data_augmentation(img)\r\n",
        "          img = vggface2_preprocessing(img)\r\n",
        "          height, width = img.shape[0], img.shape[1]\r\n",
        "          img = custom_resize(img, height, width, TARGET_SHAPE)\r\n",
        "          processed_images.append(img)\r\n",
        "        processed_images = np.asarray(processed_images)\r\n",
        "\r\n",
        "        # transform labels to categorical\r\n",
        "        for l in labels:\r\n",
        "          l = np.array(keras.utils.to_categorical(int(l.numpy()), num_classes=NUM_CLASSES))\r\n",
        "          processed_labels.append(l)\r\n",
        "        processed_labels = np.asarray(processed_labels)\r\n",
        "\r\n",
        "        return processed_images, processed_labels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nT5MifB7j4xK"
      },
      "source": [
        "## **Training**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZF2hx3dk3Rb"
      },
      "source": [
        "Creation of training and validation generator on the respective datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Md1EablSj9LA"
      },
      "source": [
        "train_generator = Generator(batch_size, tot_train_sample, train_dataset, validation=False)\r\n",
        "valid_generator = Generator(batch_size, tot_valid_sample, valid_dataset, validation=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LcaJDyVIlKNZ"
      },
      "source": [
        "Launch of the training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ySZfbRzNlB-N"
      },
      "source": [
        "history = model.fit_generator(\r\n",
        "    epochs=n_epochs,\r\n",
        "    verbose = 1,\r\n",
        "    generator=train_generator,\r\n",
        "    validation_data=valid_generator,\r\n",
        "    initial_epoch = initial_epoch,\r\n",
        "    callbacks = callbacks_list,\r\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LX-nIeW7mJLQ"
      },
      "source": [
        "## Model saving and analysis plots"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9jt-9rcImrvw"
      },
      "source": [
        "import os\r\n",
        "from keras.models import save\r\n",
        "\r\n",
        "model.save(os.path.join(log_dir,'/model/resnet50.h5'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9C1mx4fKmKBJ"
      },
      "source": [
        "%load_ext tensorboard\r\n",
        "%tensorboard --logdir=log_dir"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zwHwva8amuU2"
      },
      "source": [
        "import matplotlib.pyplot as plt\r\n",
        "\r\n",
        "# Plot training & validation accuracy values\r\n",
        "plt.plot(history.history['accuracy'])\r\n",
        "plt.plot(history.history['val_accuracy'])\r\n",
        "plt.title('Model accuracy')\r\n",
        "plt.ylabel('Accuracy')\r\n",
        "plt.xlabel('Epoch')\r\n",
        "plt.legend(['Train', 'Test'], loc='upper left')\r\n",
        "plt.show()\r\n",
        "\r\n",
        "# Plot training & validation loss values\r\n",
        "plt.plot(history.history['loss'])\r\n",
        "plt.plot(history.history['val_loss'])\r\n",
        "plt.title('Model loss')\r\n",
        "plt.ylabel('Loss')\r\n",
        "plt.xlabel('Epoch')\r\n",
        "plt.legend(['Train', 'Test'], loc='upper left')\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}