lec6-neural-network-cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 6\n",
    "## Topics\n",
    "- Neural Network\n",
    "\n",
    "**TODO:** add topics. hm maybe there is some plugin to do this quick?\n",
    "\n",
    "## Sources\n",
    "- video: https://www.youtube.com/watch?v=hd_KFJ5ktUc\n",
    "- original notes by Andrej Karpathy: \n",
    "  - http://cs231n.github.io/neural-networks-2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/hd_KFJ5ktUc?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "video_id='hd_KFJ5ktUc'\n",
    "HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{video_id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weights update\n",
    "### SGD\n",
    "- problems\n",
    "  - very slow progress along flat directions and jitter along steep one\n",
    "  \n",
    "```python\n",
    "x -= learning_rate * dx\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Momentum update\n",
    "```python\n",
    "v = mu * v - learning_rate * dx # integrated velocity\n",
    "x += v # intergrate position\n",
    "```\n",
    "- physical interpretation - ball rolling down the loss function + friction (`mu`)\n",
    "  - loss function is potential energy $U = mgh$, $F = -\\nabla{U}$ and $F = ma$\n",
    "- allows a velocity to \"build up\" along shallow directions\n",
    "- velocity becomes damped in steep direction due to quickly changing sign\n",
    "- encourage process to the consistent direction\n",
    "- overshoot - because \"build up\" velocity\n",
    "- eventually converging down\n",
    "- usually `0.5 < mu < 0.99`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nesterov momentum\n",
    "- \"lookahead\" - get `dx` from point where we are going to be by momentum ($\\theta_{t-1} + \\mu u_{t-1}$)\n",
    "$$\n",
    "u_t = \\mu u_{t-1} - \\epsilon \\nabla f ( \\theta_{t-1} + \\mu u_{t-1})\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\theta_t = \\theta_{t-1} + u_t\n",
    "$$\n",
    "$\\epsilon$ - learning rate\n",
    "\n",
    "$\\mu$ - momentum coefficient\n",
    "\n",
    "- we use rearangment in common application. \n",
    "That it would look more like vanilla update\n",
    "\n",
    "lets use\n",
    "\n",
    "$$\n",
    "\\phi_{t-1} = \\theta_{t-1} + \\mu u_{t-1}\n",
    "$$\n",
    "\n",
    "thus\n",
    "\n",
    "$$\n",
    "u_t = \\mu u_{t-1} - \\epsilon \\nabla f (\\phi_{t-1})\n",
    "$$\n",
    "\n",
    "because\n",
    "\n",
    "$$\n",
    "\\theta_t = \\phi_t - \\mu u_t\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\phi_t - \\mu u_t = \\phi_{t-1} - \\mu u_{t-1} + u_t\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\phi_t = \\phi_{t-1} - \\mu u_{t-1} + \\mu u_t + u_t\n",
    "$$\n",
    "\n",
    "we get:\n",
    "\n",
    "$$\n",
    "\\phi_t = \\phi_{t-1} - \\mu u_{t-1} + (1 + \\mu) u_t\n",
    "$$\n",
    "\n",
    "```python\n",
    "# x is \\phi here\n",
    "v_prev = v\n",
    "v = mu * v - learning_rate * dx\n",
    "x += -mu * v_prev + (1 + mu) * v\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adagrad updatet\n",
    "```python\n",
    "cache += dx**2\n",
    "x -= learning_rate * dx / (np.sqrt(cache) + 1e-7)\n",
    "```\n",
    "- idea - element-wise scale by history sum of squares in each dimension\n",
    "- problems:\n",
    "  - decay of update afterwhile - thus it doesn't work well for very long time\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RMSProp update\n",
    "come from slides (lec6) of Geoff Hinton on Coursera\n",
    "```python\n",
    "cache += decay_rate * cache + (1 - decay_rate) * dx**2\n",
    "x -= learning_rate * dx / (np.sqrt(cache) + 1e-7)\n",
    "```\n",
    "- slowly forgot previous cache what helps to keep moving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adam update\n",
    "```python\n",
    "m = beta1 * m + (1 - beta1)*dx\n",
    "v = beta2 * v + (1 - beta2)*(dx**2)\n",
    "x -= learning_rate * m / (np.sqrt(v) + 1e-7)\n",
    "```\n",
    "\n",
    "- with bias correction \n",
    "needed because m, v initilize in a zero and incorrect in begining\n",
    "```python\n",
    "m,v = ...\n",
    "for t in range(1, big_number):\n",
    "  dx = # evalute gradient\n",
    "  m = beta1 * m + (1 - beta1)*dx\n",
    "  v = beta2 * v + (1 - beta2)*(dx**2)\n",
    "  \n",
    "  # bias correction (which works in very few steps)\n",
    "  mb = m / (1 - beta1 ** t) \n",
    "  mv = v / (1 - beta2 ** t)\n",
    "  \n",
    "  x -= learning_rate * mb / (np.sqrt(mv) + 1e-7)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More updates\n",
    "[An overview of gradient descent optimization algorithms by Sebastian Ruder](http://ruder.io/optimizing-gradient-descent/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning rate\n",
    "**Should decay learning rate** because usually at the star of update we need moving fast but after while we need to slow down to converge to the minimum\n",
    "- step decay - _decay by half after few epochs_\n",
    "- exponential decay\n",
    "$$\n",
    "\\alpha = \\alpha_0 e^{-kt}\n",
    "$$\n",
    "- 1/t decay\n",
    "$$\n",
    "\\alpha = \\alpha_0 / ( 1 + kt)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Second order optimization methods\n",
    "[Newton's method in optimization](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization)\n",
    "$$\n",
    "J(\\theta) \\approx J(\\theta_0) + (\\theta - \\theta_0)^T \\nabla_\\theta J(\\theta_0) + \\frac{1}{2} (\\theta - \\theta_0)^T H(\\theta - \\theta_0)\n",
    "$$\n",
    "Newton parameter update\n",
    "$$\n",
    "\\theta^* = \\theta_0 - H^{-1} \\nabla_\\theta J(\\theta_0)\n",
    "$$\n",
    "$H$ - [Hessian matrix](https://en.wikipedia.org/wiki/Hessian_matrix)\n",
    "\n",
    "- nicer convergence\n",
    "- no hyperparameters\n",
    "- problem \n",
    "  - $H$ is squire matrix -- so we would have too many parameters `~1e6 x 1e6`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BFGS\n",
    "\n",
    "Quasi-Newton method - instead of inverting the Hessian matrix ($O(n^3)$), aproximate inverse Hessian with rank 1 updates over time ($O(n^2)$ each)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L-BFGS\n",
    "Limited memory BFGS. Does not form/store the full inverse Hessian\n",
    "\n",
    "- pros \n",
    "  - usually works very well in full batch, deterministic mode\n",
    "- cons\n",
    "  - does not transfer very well to mini-batch settings\n",
    "  - doesn't work good with randomness so don't forget to disable all sources of noise\n",
    "  - too heavy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Ensembles\n",
    "- almost always give you 2% extra performance\n",
    "### Related tricks\n",
    "- save checkpoints (on some epochs) from NN and make ensemble from them.\n",
    "- running average on `dx`\n",
    "```python\n",
    "while True:\n",
    "    data_batch = dataset.sample_data_batch()\n",
    "    loss = network.forward(data_batch)\n",
    "    dx = network.backward()\n",
    "    x += - learning * dx\n",
    "    x_test = 0.995 * x_test + 0.005 * x # use for test set. exponentially decaying of parameter x\n",
    "```\n",
    "almost always perform slightly better than `x` alone\n",
    "TODO: https://youtu.be/hd_KFJ5ktUc?list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC&t=2182"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. L2\n",
    "penalize peaky weight vectors and preferring diffuse weight vectors\n",
    "$$\n",
    "\\frac{1}{2}\\lambda w^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. L1\n",
    "$$\n",
    "\\lambda |w|\n",
    "$$\n",
    "leads $W$ vector to become sparse durting optimization. Thus we use only sparse subset of inputs and become nearly invariant to \"noisy\" inputs.\n",
    "\n",
    "We can combine L1 and L2. But in practice L2 usually gives better result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. Max norm constraints.\n",
    "_enforce an absolute upper bound on the magnitude of the weight vector for every neuron and use projected gradient descent to enforce the constraint._\n",
    "_ One of its appealing properties is that network cannot “explode” even when the learning rates are set too high because the updates are always bounded._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. Dropout (Srivastava)\n",
    "- forces the network to have a redundant representation\n",
    "- train a large ensable of models (that share parameters)\n",
    "  - subsampling of nn\n",
    "- _each binary mask is one model, gets trained on only ~one datapoint_\n",
    "- usually `50%`\n",
    "- in deep nn we usually start with small dropout and increase it on later layers\n",
    "- alternative: drop connect - instead of drop out neurons we drop out only connects\n",
    "- doesn't efficiant - _monte carlo approximation_ - do many forward passes with different dropouts masks, and avg all predictions\n",
    "- in test time \n",
    "  - we don't use dropout\n",
    "  - we should compensate fact that we drop out neurons on training time and mult activation to  `p` (`50%`)\n",
    "  - inverted dropout - actually we could mult activation to `1/p` on training time\n",
    "  \n",
    "- More\n",
    "  - [Dropout paper by Srivastava et al. 2014.](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)\n",
    "  - [Dropout Training as Adaptive Regularization](http://papers.nips.cc/paper/4882-dropout-training-as-adaptive-regularization.pdf) - _“we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix”._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. Drop Connect\n",
    "[more](https://cs.nyu.edu/~wanli/dropc/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. Bias\n",
    "regularizing the bias rarely leads to significantly worse performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularization. Other\n",
    "- Per-layer regularization -- rarely used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss function\n",
    "### Large number of classes\n",
    "can use **Hierarchical Softmax** [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546). The hierarchical softmax decomposes labels into a tree. The structure of the tree strongly impacts the performance and is generally problem-dependent.\n",
    "\n",
    "### Attribute classification\n",
    "build a binary classifier for every single attribute independently. Then sum loss functions of all attributes (j)\n",
    "$$\n",
    "L_i = \\sum_j{max(0, 1 - y_{ij}f_i)}\n",
    "$$\n",
    "or logistic regression classifier for every attribute independently\n",
    "$$\n",
    "P(y = 1 | x; w; b) = \\frac{1}{1 + e^{-(w^{T}x + b)}} = \\sigma (w^Tx + b)\n",
    "$$\n",
    "\n",
    "$$\n",
    "L_i = \\sum_j{y_{ij} log (\\sigma(f_i)) + (1 - {y_{ij}) log (1 - \\sigma(f_i))}}\n",
    "$$\n",
    "where $y_{ij}$ are assumed to be either 1 or 0\n",
    "\n",
    "$$\n",
    "\\frac{\\delta L_i}{\\delta f_i} = y_{ij} - \\sigma(f_j)\n",
    "$$\n",
    "\n",
    "### Regression\n",
    "_ it is common to compute the loss between the predicted quantity and the true answer and then measure the L2 squared norm, or L1 norm of the difference._\n",
    "- the L2 loss is much harder to optimize than a more stable loss such as Softmax\n",
    "- the L2 loss is less robust because outliers can introduce huge gradients\n",
    "- the L2 is more fragile and applying dropout in the network (especially in the layer right before the L2 loss) is not a great idea.\n",
    "- when it possible prefer to use bins than regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient checking\n",
    "it is better to use _centered difference formula_ because it gives error terms on order of $O(h^2)$ in comparison commont approximation $O(h)$\n",
    "\n",
    "$$\n",
    "\\frac{df(x)}{dx} = \\frac{f(x+h) - f(x-h)}{2h}\n",
    "$$\n",
    "\n",
    "- compare relative difference\n",
    "$$\n",
    "\\frac{|f_1' - f_2'|}{max(|f_1'|, |f_2'|)}\n",
    "$$\n",
    "\n",
    "- `relative error > 1e-2` usually means the gradient is probably wrong\n",
    "- `1e-2 > relative error > 1e-4` should make you feel uncomfortable\n",
    "- `1e-4 > relative error` is usually okay for objectives with kinks. But if there are no kinks (e.g. use of tanh -nonlinearities and softmax), then 1e-4 is too high.\n",
    "- `1e-7 and less` you should be happy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Recommendations\n",
    "- use double precision float point \n",
    "- don't hit float point limits with very small $df$, what may required up scalling of learning rate. ideally on order of `1.0`, _where your float exponent is 0._ [\"What Every Computer Scientist Should Know About Floating-Point Arithmetic\"](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)\n",
    "- crossing kink of objective function (for `ReLU` it is `0`) we could get significant difference on gradient checking, so we should check whether \"winner\" was changed between $f(x-h)$ and $f(x+h)$.\n",
    "- perform grad check only for few data point - faster and we less likely get case with kink\n",
    "- don't use too small $h$ [a chart that plots the value of h on the x-axis and the numerical gradient error on the y-axis](https://en.wikipedia.org/wiki/Numerical_differentiation#Practical_considerations_using_floating_point_arithmetic)\n",
    "![](https://upload.wikimedia.org/wikipedia/commons/thumb/4/41/AbsoluteErrorNumericalDifferentiationExample.png/600px-AbsoluteErrorNumericalDifferentiationExample.png)\n",
    "- drop few first iteration before loss function starts to fall down (because random initialization could falls in pathological edge case\n",
    "- check without regularization first because it also contribute to loss function (regularization loss)\n",
    "- turn off drop off and augmentation\n",
    "- check only few dimensions but for all parameters\n",
    "- check loss function on init, for softmax it could be -ln(1/classes)\n",
    "- _increasing the regularization strength should increase the loss_\n",
    "- **try to overfit** check algorithm on small portion of example we should get 0 loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## History of Convolutional Neural Network (CNN)\n",
    "- 1980 Fukushima. **Neurocongnitron**\n",
    "- 1998 LeCun, Botton, Bengio, Haffner. **LeNet-5**. Gradient-based learning applied to document recognition\n",
    "- 2012 Knizhevsky, Sutskever, Hinton. **AlexNet** + Relu. ImageNet Classification with Deep Convolution Neural. Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Visualization\n",
    "### Loss function\n",
    "- sometimes use log graph\n",
    "- for funny cases https://lossfunctions.tumblr.com/\n",
    "### Accuracy\n",
    "- train and validate data set\n",
    "### Ratio of weights:updates\n",
    "### Activation/gradient distibution histogram\n",
    "### 1st layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}