3.7 Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/punitarani/MAT-494/blob/master/3.7%20Neural%20Networks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.7 Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A neural network is a collection of connected layers of nodes to loosely model the neurons in the brain.\n",
    "\n",
    "### How does a Neural Network work?\n",
    "\n",
    "- The input layer receives the input data.\n",
    "- The hidden layers transform the input data into the output data.\n",
    "- The output layer produces the output data.\n",
    "- The weights and biases are updated using the backpropagation algorithm.\n",
    "- The loss function is used to measure the performance of the model.\n",
    "- The loss function is minimized using the gradient descent algorithm.\n",
    "- The model is tuned using the hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## High-Level Model\n",
    "\n",
    "$ \\text{Input} \\rightarrow \\text{Hidden Layer} \\rightarrow \\text{Output} $\n",
    "\n",
    "$ \\text{Input} \\rightarrow \\Sigma f(x) \\rightarrow \\text{Output} $\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neuron (Perceptron)\n",
    "\n",
    "A Neural Network consists of Neurons, which are the basic building blocks of a neural network.\n",
    "A Neuron is a single node in a neural network.\n",
    "A Neuron is defined as:\n",
    "\n",
    "$y_k = \\varphi(\\sum_{i=1}^n w_{ki} x_i + b_k)$\n",
    "\n",
    "where:\n",
    "- $y_k$ is the output of the $k$-th perceptron.\n",
    "- $x_i$ is the input of the $i$-th perceptron.\n",
    "- $w_{ki}$ is the weight of the $k$-th perceptron.\n",
    "- $b_k$ is the bias of the $k$-th perceptron.\n",
    "- $\\varphi$ is the activation or transfer function.\n",
    "\n",
    "\n",
    "### Activation Function\n",
    "\n",
    "The activation function for a neuron is used to transform the input data into the output data.\n",
    "The activation function is defined as:\n",
    "\n",
    "$u = \\sum_{i=1}^n w_i x_i + b$\n",
    "\n",
    "$y = \\varphi(u)$\n",
    "\n",
    "where:\n",
    "- $u$ is the weighted sum of the inputs.\n",
    "- $y$ is the output of the neuron.\n",
    "- $x_i$ is the input of the $i$-th neuron.\n",
    "- $w_i$ is the weight of the $i$-th neuron.\n",
    "- $b$ is the bias of the neuron.\n",
    "\n",
    "#### Step Function\n",
    "\n",
    "The step function is a simple activation function that returns 1 if the input is greater than 0, otherwise it returns 0.\n",
    "\n",
    "$y = \\begin{cases} 1 & \\text{if } u > 0 \\\\ 0 & \\text{otherwise} \\end{cases}$\n",
    "\n",
    "#### ReLU Function\n",
    "\n",
    "ReLU stands for Rectified Linear Unit.\n",
    "It is one of the most popular activation functions.\n",
    "ReLU performs a linear transformation of the input data.\n",
    "\n",
    "$y = \\begin{cases} u & \\text{if } u > 0 \\\\ 0 & \\text{otherwise} \\end{cases}$\n",
    "\n",
    "#### Sigmoid Function\n",
    "\n",
    "The sigmoid function is a smooth activation function that returns a value between 0 and 1.\n",
    "\n",
    "$y = \\frac{1}{1 + e^{-u}}$\n",
    "\n",
    "#### Softmax Function\n",
    "\n",
    "The softmax function is a smooth activation function that returns a probability distribution.\n",
    "\n",
    "$y_k = \\frac{e^{u_k}}{\\sum_{i=1}^n e^{u_i}}$\n",
    "\n",
    "It is often used in the final output layer, especially with classification problems.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
    "        x = torch.flatten(x, 1)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "net = Net()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "torch.Size([6, 1, 5, 5])\n"
     ]
    }
   ],
   "source": [
    "params = list(net.parameters())\n",
    "print(len(params))\n",
    "print(params[0].size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.1158,  0.0415,  0.0989,  0.1751, -0.0513,  0.0245,  0.1228, -0.0626,\n",
      "          0.1342, -0.0330]], grad_fn=<AddmmBackward0>)\n"
     ]
    }
   ],
   "source": [
    "input = torch.randn(1, 1, 32, 32)\n",
    "out = net(input)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.zero_grad()\n",
    "out.backward(torch.randn(1, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(1.0853, grad_fn=<MseLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "output = net(input)\n",
    "target = torch.randn(10)  # a dummy target, for example\n",
    "target = target.view(1, -1)  # make it the same shape as output\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "loss = criterion(output, target)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MseLossBackward0 object at 0x000001FDED70A680>\n",
      "<AddmmBackward0 object at 0x000001FDED709E70>\n",
      "<AccumulateGrad object at 0x000001FDED70A680>\n"
     ]
    }
   ],
   "source": [
    "print(loss.grad_fn)\n",
    "print(loss.grad_fn.next_functions[0][0])\n",
    "print(loss.grad_fn.next_functions[0][0].next_functions[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backpropagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv1.bias.grad before backward\n",
      "tensor([0., 0., 0., 0., 0., 0.])\n",
      "conv1.bias.grad after backward\n",
      "tensor([ 2.3788e-02, -2.7721e-03,  8.8365e-03,  5.4275e-03, -1.7853e-06,\n",
      "        -2.6267e-02])\n"
     ]
    }
   ],
   "source": [
    "net.zero_grad()\n",
    "\n",
    "print('conv1.bias.grad before backward')\n",
    "print(net.conv1.bias.grad)\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "print('conv1.bias.grad after backward')\n",
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.01\n",
    "for f in net.parameters():\n",
    "    f.data.sub_(f.grad.data * learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
    "\n",
    "optimizer.zero_grad()\n",
    "output = net(input)\n",
    "loss = criterion(output, target)\n",
    "loss.backward()\n",
    "optimizer.step()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}