Hyperparameter Tuning TF and ACI deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bf74d2e9-2708-49b1-934b-e0ede342f475"
    }
   },
   "source": [
    "# Training, hyperparameter tune, and deploy with TensorFlow\n",
    "\n",
    "## Introduction\n",
    "This tutorial shows how to train a simple deep neural network using the MNIST dataset and TensorFlow on Azure Machine Learning. MNIST is a popular dataset consisting of 70,000 grayscale images. Each image is a handwritten digit of `28x28` pixels, representing number from 0 to 9. The goal is to create a multi-class classifier to identify the digit each image represents, and deploy it as a web service in Azure.\n",
    "\n",
    "For more information about the MNIST dataset, please visit [Yan LeCun's website](http://yann.lecun.com/exdb/mnist/).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://docs.microsoft.com/en-us/azure/machine-learning/service/media/overview-what-is-azure-ml/aml.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get started. First let's import some Python libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) \\n[GCC 7.3.0]'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-11-13 09:20:23.462389\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbpresent": {
     "id": "c377ea0c-0cd9-4345-9be2-e20fb29c94c3"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import os\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "nbpresent": {
     "id": "edaa7f2f-2439-4148-b57a-8c794c0945ec"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure ML SDK Version:  1.0.72\n"
     ]
    }
   ],
   "source": [
    "import azureml\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# check core SDK version number\n",
    "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize workspace\n",
    "Initialize a [Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the existing workspace you created in the Prerequisites step. `Workspace.from_config()` creates a workspace object from the details stored in `config.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workspace name: azuremlservice\n",
      "Azure region: westeurope\n",
      "Subscription id: 70b8f39e-8863-49f7-b6ba-34a80799550c\n",
      "Resource group: azuremlserviceresourcegroup\n"
     ]
    }
   ],
   "source": [
    "ws = Workspace.from_config()\n",
    "print('Workspace name: ' + ws.name, \n",
    "      'Azure region: ' + ws.location, \n",
    "      'Subscription id: ' + ws.subscription_id, \n",
    "      'Resource group: ' + ws.resource_group, sep = '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "59f52294-4a25-4c92-bab8-3b07f0f44d15"
    }
   },
   "source": [
    "## Create an Azure ML experiment\n",
    "Let's create an experiment named \"tf-mnist\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "nbpresent": {
     "id": "bc70f780-c240-4779-96f3-bc5ef9a37d59"
    }
   },
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "script_folder = './tensorflow-mnist'\n",
    "os.makedirs(script_folder, exist_ok=True)\n",
    "\n",
    "exp = Experiment(workspace=ws, name='tf-mnist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "defe921f-8097-44c3-8336-8af6700804a7"
    }
   },
   "source": [
    "## Download MNIST dataset\n",
    "In order to train on the MNIST dataset we will first need to download it from Yan LeCun's web site directly and save them in a `data` folder locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./data/mnist/test-labels.gz', <http.client.HTTPMessage at 0x7f2e00b0c3c8>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib\n",
    "\n",
    "os.makedirs('./data/mnist', exist_ok=True)\n",
    "\n",
    "urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz', filename = './data/mnist/train-images.gz')\n",
    "urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz', filename = './data/mnist/train-labels.gz')\n",
    "urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz', filename = './data/mnist/test-images.gz')\n",
    "urllib.request.urlretrieve('http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz', filename = './data/mnist/test-labels.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "c3f2f57c-7454-4d3e-b38d-b0946cf066ea"
    }
   },
   "source": [
    "## Show some sample images\n",
    "Let's load the downloaded compressed file into numpy arrays using some utility functions included in the `utils.py` library file from the current folder. Then we use `matplotlib` to plot 30 random images from the dataset along with their labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "nbpresent": {
     "id": "396d478b-34aa-4afa-9898-cdce8222a516"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA44AAABBCAYAAACeofpoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2deXxMV//HP+fOxBIkdknsUkoFQRBbmqi9POkjtqJaa2spqkirWvVD0aIeWWyxVGupXZUqglD7vhNbmoiGRBaRzSTz/f0xM/eZSWYmy9w7Nw/n/XrdV5J7b+7nnHvP9j3L9zAiAofD4XA4HA6Hw+FwOJYQlA4Ah8PhcDgcDofD4XCKN9xw5HA4HA6Hw+FwOByOVbjhyOFwOBwOh8PhcDgcq3DDkcPhcDgcDofD4XA4VuGGI4fD4XA4HA6Hw+FwrMINRw6Hw+FwOBwOh8PhWIUbjhwOh8PhcDgcDofDsYpshiNjrBpjbDljLIYx9pIxFs0Y+w9jrLyMmg0YY//HGDvNGItnjKUyxi4zxr5ijJWRS1evLTDGPmOM3WaMZerjvUhuXb12RcbYQsbYPb12PGPsCGOso0x6irxnxti3jDGycmhk1C7LGJvOGLumj28CY+wkY+wjxhiTUfdLxthWxtgDfRyj5NIyo23vdPUmY2wDY+wWYyyFMZauz0+LGWOucmgaadv9+ypVZiiVjxTMQ4qVzbnC4cgYe6h/x8EyaSiWh/T6di0z9JqW0vELuTT1unZPVwqXkXavi5hybQ0l6127p2emQHvdQjjsUUa+dnaC1GlKLXUAAYAxVhXAGQBuAFYAuA7AA8AYAD6MsfZElC6D9HAA4wD8BmADAA0APwBzAPRnjHkTUYYMugDwI4AJAHYCWASgkf7v5oyxzkSklUOUMVYbwFEAZQGsBhAJwBlAUwDV5dCEcu95B4B7Zs43BTAVwB4ZNMEYEwD8AaAdgJ8ABAFwBPA+gLXQfetAObQBfAcgEcBFAHYrxBVKVzUAuEKXhx4ByAbQBMBoAAMZY55E9FRqUQW/ryJlBhTIRwrnIaXec27+D0BlmTUUyUOAYmWGgeMAVuY6J1tHoh4l0pVi3xfK1EVKtTUUqXeNsFt6VrC9bg57lJGvnZ2gR7o0RUSSHwCWACAA7+c6/77+/AyZdL0AOJs5P0evO14m3cYAtAC25zr/qV53kBy6eo3jAGIAuMqlUVzes5XwrNDrvivT89vqn/9jrvMlADwAkCxj3OoZ/X4dQJSd3qnd05WVsPTTv/9pr8r3VbLMsBIm2fKRUnmouLxnAC2ga+RP1usG2/nbypqH9BqKlBn6eK2zs2axSFd2/r52r4sUbNMpUu/q9eyanqFQe91MOOxSRiqYppS0EyRNU3JNVfUDkAFgc67zvwLIBDBMDlEiOk9EKWYu/ar/6SGHLnQZjEGXAY1ZBSAdwBA5RBljPgA6APieiP5hjDkwxhzl0DJGwfecB318BwKIBbBfJhkn/c/HxieJ6CWABABpMumCiB7I9WxLKJWurPC3/mcFmZ6vxPdVpMywhB3ykVJ5SPH3zBhT6fX2QzfaqwSy5qHiUGYwxkowxsraSU7xdJULuctIReoipdoaSsQ1N3ZMz4q0142xZxn5utkJxkiVpuQyHEsCyCS9qWuAdMOwGQDqMcbkHo42pob+5xOZnt8Kup6Es8YniSgTwGX9dTnoqf8ZzRjbA927TWOMRTLG7F1xAfK/Z3P0h65RupaIcmTSOAsgGcA0xlg/xlgt/VqTeQBaAvhWJl2lUDRdMcZKMcYqM8ZqMMa6QjcSBgD7ZJJU4vsqVWZYQu58pFQeKg7v+TMADQGMt4MWAEXykNJ1UV/oGl+pjLGnjLEgxpizjHqKpisFvm9xQ4m2hj2xZ3ouDu11u5eRZnhV7QQD0qUpmYZFt0M3NOqZ67yn/jwBaCHXsGwuTRWAU9DN5X1TJo1rAJ5YuLZFH98SMuju1D/7KYATAAZDN3/7uv78MHu8Y3u9Zwu6x6HLjHVl1ukI4I5R+iUAzwG8Z8e42mt6kKLpCrrKw/g9PwQw+FX6vkqVGVbCI3s+UiIPKf2eAdSFbjQ1UP93Hdhhqqq985CSZQZ067OmAHgPwFDoRk4IwFUAZV/RdGX3MjKXvl2nb+bStmtbw95xtXd6hsLtdaXKSHunKSXLDKnTlCzOcaAbin0PwBbG2CToMl5j/XkNAAfoHCPYgyUAvAFMJ6I7Mmk4AsiycC3T6J6XEuuW0/9MBeBHumlfYIzthG7d0HeMsZ/IPs4f7PGeTWCMvQnd9KhwInoos9wL6NLxbwBOAqgI3QLrjYwxfyI6KLO+PVE6Xe0CcBs6JxvNAfwLQBWZtAzY+/sqVWbkwY75SIk8pPR7XgZdo36xTM+3hL3zkGJlBhG1yXVqPWPsKoC5ACbqf0qN0ulKiTKyuGD3toY9USA9K91eV6qMNOZVthOkT1MyWvD9APyD//ZYZANYDt38ZQLQ1A69CLP1Witk1lFqxHGP/tlzzFz7SX+t0avyns3ofq/XHSizThPopmx8kuu8I3SNoigAKjvE114jjsUiXRlpNoWuwP3yVfm+SpUZFvRkz0dK5SEl3zN0a1a0ADoYnasDZZzjyJ2HiluZ4aCP70mZnl9s8q89vq8ZPUVGHJVoaygV11xhkDs9K9JeLw5lpL3SVDEsM4qcpmTbx5GItkI3Z7g5AB8AbkT0if5cNsy7g5cMxti3AGZA5+r9Ezm1oHP4UJkxVtLMteoAEkjfAysxj/Q/48xc+0f/U7bF8oDd37Oxrhq6IfdE6KZJyclnAEoB2Gp8knQuqvcCqA1dYfeqoHi6MoaIrgK4BGCsTBJKfF+lygwT7JiPlMpDirxnvd5i6NacxTHG3mCMvQFdPAHAWX/OLq7+7ZCHiluZoYH+28skUSzyrwE7fF/FUaqtURyQOz0r0V4vDmXka2InmMWWNCWb4QgARJRDRJeJ6DgRPWWMuUCXMCNIxn1hGGMzAcwEsB7ASNKb1zJyDrp32TpXOEpBN0/8vEy6hkW2NcxcM5yTa08nJd6zMb0BVAPwMxFZGv6XCsMeZCoz19S5fr4KKJquLFAauqmNcqDE91WqzMiNvfKRUnlIqfdcGrqpg+8CuGt0HNVfH6L/e6RM+pbCJFceKlZlhv771oB8ji6KS/41Rs7vqygKtzUUxw7pWYn2uqJl5GtkJ5jFljQlq+FojH4D6KXQNRzkWHNg0PkGOg99P0O3IN8e6/t+hW6YeVKu86Ogm4q1QSbdXdCtKRli7GKXMeYK3Zz1u0Qky8iuQu/ZmBH6n6vtoHVT//Mj45P6njB/AEkA7tshHPZCkXSlr6jMnfeDzkX2aak19SjxfZUqM3Jjr3ykVB5S6j2nQTf9K/dhGBHar//7NylFFcxDSpUZlSxcmg1dR8QeqTX1KJKuFPy+ilEM2hp2Q8H0nDsc9mivK1JGAq+XnSBHmmJyGNn6iuMsdFOfHgJwhm4Pk5YAviKi7yQX1emOAxAMIBrA19DNnTbmCcnkxIQxFgSdp7Od0A29NwIwAToPc53kSpiMsdHQueK+AWANdBtqjwHgCqAXER2QQVOx96zXd9NrX6C8i37l0KsN4CJ0U602QPdNK0KX4esAGEdEoTJpf4D/Tt34FLrvu0j/999E9LNMukqkq5365x+Gbl+yUtCVGQOhcyPtS0SXZdBV5PsqVWYY6dstHymchxR9z7nCUge6OjGEiCR3Pa9UHtJrK1Fm/AidU4sj0KXlstBtDeIHnSdBPyLKkFpXr233dKXw97V7XaRUW0PBetfu6Vmp9rqV8NSBvGXka2UnyJKmZFp0WQI6d68PofMWlAjgTwDd5NAz0l0HUxfVuY+jMmqrAHwOnbv5LOg20l4MmdyB59LuA11PYxp0vb4HALR/Fd+zXn+6XmeU3O/WSNMdOicPj6DzNPYcwDEAfWTWParge7Z3uuoP3Xq3GH25kQGd58AgALVete+rZJmh17drPlIwDyn6nnOFpQ5kdPygZB7S69u7zPCHrm0Rq49vGnR7ok0HUOpVS1cKl5F2r4ugUFtDibjqde2enqFQe91KeOQuIxVJU3ptJcoMydOULCOOHA6Hw+FwOBwOh8N5dbDbGkcOh8PhcDgcDofD4fxvwg1HDofD4XA4HA6Hw+FYhRuOHA6Hw+FwOBwOh8OxCjccORwOh8PhcDgcDodjFW44cjgcDofD4XA4HA7HKtxw5HA4HA6Hw+FwOByOVbjhyOFwOBwOh8PhcDgcq6gLdbOjMzlVcZMrLACApOjbCURU5XXQFdLikZCQwHJrOpSvhtIOKtl0LcWV63JdOXRfxbzLdbnuq6TL6yKu+zrovop5l+tyXXvqAgUwHBljowGMBoCSLm+gy/Q1MgTvv2z5pN3fr4vu/VWfwpzmG6OD8Jark2y6luLKdbmuHLqvYt7lulz3VdLldRHXfR10X8W8y3W5rty6eSCiAh8VajUkuQFw/nXRbdmyJZnT7L/8pKy6luLKdbmuOcLCwggATZs2rUi6csN1uS7XtQ1eF3Hd4qrbo0cPAkC+vr704MEDSkxMpPj4+CLpyg3X5bqvui4R8TWOrxJHjhyBIAj4+OOP7aL3zz//wM/PD4Ig4K233sKpU6fsosuxH9euXcOYMWPAGEPfvn2VDg6Hwymm7NmzByqVCgEBAbh7967SweG8Iuzfvx+MMURERMDd3R0NGjTA119/jaysLKWDxvkf58mTJwgNDUW5cuXg7++PsLAwZGdnKx2sYs8rYTjeuXMHjLF8j4CAAFnDsXnzZjDGoFKpoFKpUKVKFWzevFlWTQDYsWMHVCoVOnfuDMYYSpUqJbmGVqvFypUr0ahRIzF+1atXx7Fjx1CtWjU8ffoU7du3R506dZCeni65vpLs3r0btWvXFtNR7dq1kZCQYDf95ORkODg4iEd8fLxddHft2oVmzZqhXLlyuHfvHlq1aiW5RlZWFlxdXcU0pVKp4OrqiqNHj0quVRxwc3Mz+ZaGQ6VSwcHBAdOmTcOff/6pdDA5/2PcvXsXpUuXhiAIJkfTpk2xdetW2RvZ//zzD/7973+DMYY//vjDro2v8PBwk3rXcISHh+P06dOyams0GkRHR8PLywtlypRBgwYN8Pz5c1m0Ro4cKX7XGTNmyKJhDUMZlZiYaFddrVYrHqmpqfD398eKFStQqlQpNG7cGDk5ObJpX758GQ4ODnj8+LFsGrmJiYnB5s2bMWPGDCxduhRLly6FIAjw8vJ67TpkwsPDoVKpJH3/RIQrV67A1dUV/v7+GDJkCFJTUzFlyhR8/PHHKFmyJL755hvJ9Iz5/PPP85RTKpUKLi4u2LhxoyyaBp4/f44PP/xQLCsnTJiAzMzMIj3rlTAcw8PDTf4OCQkRh1RDQkLQp08fADoDKzQ0VLZwDBkyBIIgiAZGUlIShgwZgnr16uHq1auyaEZGRoojQQMHDsTGjRvx/fffS66Tnp6OMWPG4M6dO+K5SZMmYfPmzYiKikJUVBQaNmyImJgYLFy4UHJ9Jdi1axfc3NzQp08fxMTEiN81JiYGrVq1spsBl5sqVfKsVZace/fuYf78+RAEAUFBQahXr54sOrdu3cLTp09Nzj19+hQ9evTAsmXLZNG0RkJCAmJjYxEbG4u0tDRJnx0WFpZvo/LHH3/EoEGDsGPHDkm1czNgwAAIgoBatWrJqmPM1atXsWTJEty7dw/3799HaGgoAgMDERgYiGnTpmHOnDl2CwsAZGZmIiMjAy9fvrT5Oe+99x4EQbB7o9pA/fr1sXDhQvTr1w916tSBg4MDGGO4fv06BgwYgDfffBMPHz6URTsjIwOenp7i33PmzEGjRo1k0TJGq9UiOTkZ3bt3z2MwC4KArl27omfPnjh//rws+gcPHoSPjw9q164NV1dXnDp1Co8ePcLPP/8sqU5OTg7Gjx+PtWvXinXQggULULduXUl1/hcoU6YMli9fjoULF4Ixhlu3bmHDhg2y6X377bfo27cvXFxcZNMAdHVeWFgYOnXqhLp162Lw4MGYP38+PvvsM0yePBmMMVy+fFlWw1Gr1WL37t0YMGAAatSogd69e2Pp0qWy6eXHs2fPEBgYCCKCm5t0TmC0Wi1atGiBp0+fom3btuJAi7u7OwDA2dkZvXr1kkzPmIkTJ5o9Hx8fj6lTp0pef2RkZODBgwe4fv06KlasiA0bNsDFxQUnTpzAoEGDoFIVzUGVLIZjWloaHj9+LBpRgiCgSZMmOHbsmBxyIgaDcezYseK5sWPHYvv27aLxKAeGj2OJv//+G9999x3i4uIk1zYYjS4uLggJCcGAAQNQsmRJyXUAXU8NADg5OSElJQWLFy9Gv379ULJkSZQtWxYLFiwAESEsLMwmnZycHDx8+BCzZs0SG5b+/v6oUqUKAgMDce3aNVmNtkePHuHTTz9FQEAA4uLiQETYuXMnMjMzkZmZiSdPnmDkyJF5GpzG6U5KAgMDxd9btGghi0ZuFi9ejLNnz2LMmDEYNGiQbDo1atRAnTp1xL9HjBiBatWq4eXLl9ixYwc0Go1s2sYcOnQIPXr0QL169VCzZk3UrFkTvr6+SE1NlUzj1KlTBRr5ef78OcLDwxEfHy9Zb3pSUhIeP36M4OBgDB06FFu3bgVjLP9/lJAlS5YgODgYDRo0QP369TF+/Hj88MMP+OGHH7Bw4UKEhITIpv3ixQtkZGQgKysLGo0GS5cuhYuLCxwdHeHn54eMjIwiPzstLQ179uzJ932mp6fjyJEjaNKkCaZNm2azwZqbcePG4ddff8WDBw+QlZWFEydOoF27dqhfvz6io6PRqVMnWUYec3Jy8OzZM/Hv8ePHS65hzLlz5xAaGooFCxaYdKJVqlQJQUFB4hESEoKVK1fCy8tL8jDExMSga9euyMjIQPfu3REREYG0tDQMGjQIFy5ckEwnJSUFXl5eeTrRtFot4uLiFOu8VBK1Wo0JEyZg+PDhACBbh1N2djYyMzPx66+/yjLi+PLlS8TExGDGjBnw9PTE6NGjERERAU9PT8yZMwe//fYb9uzZg1u3biEnJwePHz9Gz549JQ8HoItrnz590KdPH2zduhXlypVDZGQkPvvsM9na7Y8ePcLy5cstXn/8+DGuXLkiW8fAggULsGDBApQoUQLx8fFo27YtAN13SUpKkkXT3Gy8N998EwAQFxeHWbNmSdrmmThxIho0aIBmzZqhb9+++Pjjj3H27Fl4e3vD29sbDg4ORXuwuYWPlo6CLMY8dOgQubu7kyAINHnyZJo8eTINHz6cBEEgFxcXev78eaEXY0qxCDQkJIQAUEhIiOS6Fy9eJEdHR1KpVCQIAqlUKpPDcK5Tp04m/2erQ4KkpCQSBIHc3d3p5MmCLza3FFdrujk5OfTixQuKiIig9PR0s/esXbuWBEEgZ2dns4vXC6KbkpIiOmOxdpQrV44mTZpECQkJksY3JiaGKlWqRIwxYoxRs2bNaNeuXVaff+vWLRo0aBCp1Wq6fft2kXStERAQQGq1mtRqNR0+fDjf+6XQZYzR6NGjLX7rwujmR2ZmJqWlpVFaWhpptVrKyMig/v37kyAItHr1atl0Ddy8eVP83vXq1SMPDw/y8PCgypUrU6lSpejp06dERDRhwgTq379/kXRjY2PJ19dX/I65D0EQzJ43l74Lo7tq1Spq3749VahQgRhjYv7ctGkTff7551SzZs0Cvydb3vM///xDNWvWFN9z//79afjw4dSkSRPxnK+vr+S6REQajUb8tr6+vhQQECBqli9fnurXr09r164tsm5CQgIJgkCCINCzZ88shuPy5cvifYGBgfTy5UtZ4pubW7duifG9ePGi2XuKWhdlZWVRp06dSBAE8vPzo19++aVQYStsWfXgwQOqUKFCnvySnJycb9vCFl1jYmNjqXPnzvTVV19RVlYWERFt2bKFzp8/TyEhISQIAkVFRdmsGxcXR507dxbTTK1ateibb74R37dKpaK9e/fKHl8DhnduLY3LoWuJ69evi+m6oLqF4c6dO8QYI29v7wLXg4XRbdmypUn7cMaMGbRlyxaL5YJUugays7NJq9VSREQEeXt7U2BgIMXGxlJiYiJlZWVRVlYWtWnThhYtWiSpLhHR8+fPqVu3bma/nVarpU2bNlGTJk3Iw8NDrH+l0DU8PyUlxeRckyZNxHwmCAKdOnXK7P/aohsZGUlVq1YVNVavXk2xsbGUmppK169fp1q1almsQ4qq26NHDzGNZWdnFyic+ekSUeH2ccyP+Ph4dO/eHQAwY8YMzJo1SzROGzVqhMDAQPj4+CAiIgJOTvK5YjZH7umsUlK1alVUrVoVMTExJucnT56Mli1biiM2ERERkuoaenbXrFkj9pbIhSAIKFOmDHx8fMxej4mJwdSpUwHoeuATExNRuXLlQus8fvwYI0eONDnXqFEjvPXWWwB0UzqOHz+O1NRULFmyBHfv3sXu3buLPOSem5YtWyIxMRGenp6YOXMmunXrZnHN6O3btzF79mzs3r0bWq0WSUlJKFu2rCThMJCYmIgHDx5Aq9UC+O+or5xcu3YNnTt3xpIlS1C6dGnZ9XKPkGs0GlSqVAkAbBoJKgjHjx/HJ598ggMHDqB69eqoV6+eGJ6IiAgMHjwYv/32GwYOHIjNmzfj5MmTRdIJDw/HX3/9Vej/W7BggU1Tz8uWLYtKlSph2LBhICK0bdsWfn5+qFy5cp7ySi7CwsLw2WefiVN/09LSULJkSURFRaF169YAgLZt22LPnj2y6D958gQA8PDhwzzTNYcOHYp58+bB0dGxyM/fvXt3ge4zXsMyf/78IusVBiIqcpotCFu3bsXRo0fBGEPz5s0xePBg2bQAYOnSpXlmARw4cADOzs6y6hqj0Whw6NAhbN26FSVKlAAA9OvXDwBQq1YtjBs3DlOnTsWWLVts0lm2bBkOHz4MQLc84cSJE+IUQkBXbso1AmUOQx1UXLCXL4UGDRrIUg9evHgRjDHMmTMHEydOtKkMKgxEhG+//RY3b97EwoUL4efnB2dn5zzODU+cOIGbN2+iffv2kuofPXoU//73v5GSkmJ2CUxCQoLYZo6OjpZ8aQ5jDE5OTkhOTsaWLVtw5coV3Lx5U7zeoUMHsV6SipycHLRu3VpcqrJ69WoMHToUgqCb9Nm4cWN06NBBcp8o+/btw6RJk7B06VLJ2shAAfZxLAwBAQHQarWYOHGiaDQCuqFfw/SNq1evonfv3pIbUda4c+eObGuGYmJi8J///AfR0dEAAB8fH3Tv3h2ffPKJWJk1b94cbdq0kXTR/O+//47Nmzdj7ty5Fo05exETE4PWrVvj2bNnYIyhW7duaNCgQZGe5e7ujtOnT2PDhg2oV68ehg4dCkdHR9F402g0uHz5MiZNmoSTJ09i7969iIyMlGRNzfbt2xEfHw9PT0/88MMPeOeddyze+9dff6FHjx5IS0uDo6Mjjhw5IrnRCOg8yl27dg2CIKB06dIoU6aM5BrGpKenY9iwYVi/fr1djEZjtFotfvnlFyxbtgxnz54FAFkc8hh48eIFxo0bh9jYWHTq1EksxA107NgRo0aNwujRozF16lTs3LlTXAdRGHJycsxOeRUEQZwCde/ePfTo0SPPfbauWR44cCAGDhxo9po9OiEAYPTo0Wjbti1q1qyJLVu2oHTp0jh27Bj69euHxMRENG3aFAcOHJAlbaekpOSpMF1cXMRlA0FBQbh7967oEbQoGBr3+bFv374iPd8WNm7cKHbEGZYUSMWDBw+wcuVK8W+5jcbY2Fhs3rxZdJRiwM/PT1ZdY5KTk9GmTRtUqlQJ5cuXt3hfx44dbda6cuUKAKBixYo4d+4cqlSpgsjISDEd5e5glRvD2tHigqHDxtDJKDW//vorAJ3vCjkwrFf98ssvZXm+JVJSUrBu3Tps2rRJ1DZeDmO4Z8yYMahevTqaNm0qmfadO3cwefJkpKSkoHHjxlizxnQfwqysLPTv3x8AxLafHGi1WrRr187EZwdjDG5ubli4cKHk6TwyMlJs/0+ZMgX9+/e3W17y9vZGUFCQpM+ULOQajUb8CMYZwbDQ1rj3zVJDRi7kGG3UaDQ4deoUmjdvjiVLloAxhqNHj+KPP/5AYGCgSQ9ogwYN4O/vL9maonnz5mHIkCEgInh6eiI7OxsXLlzA8uXL4enpidDQUISFheHhw4eyNg4zMjKwbt06tGjRQuzV79KlC7Zt21bkZzo4OKBNmzZYunQpJk2ahIoVK5qM+Dk4OKBVq1bYvHmzaNhs2rTJtojoMazTCQgIsGg0HjlyBJ6enujZsyfUajWmT5+O6OhotGnTRpIw5OaDDz4Qf+/evbtsOgamT5+OixcvomHDhuK5jIwM/Pjjj5gwYQIuX74siy4RYf/+/Rg2bJhoNPr6+qJ58+ay6AG60cZr165h9+7dZgvx33//HbNmzQIRwd/fH2+//XaRdKKioswuiq9SpQoqVaqESpUqoU2bNti/fz9q1KhRJI2isHr1arvo/PHHHwgPD8eKFSsAAL1798Z7772H+Ph4fPzxxzhx4oQsRmNERASaNm1q4lhhw4YNuHz5Mvr164d69eqBiFC/fn2b1jA1adKkQPfJ2QmSm/T0dIwYMQIffviheG779u2oX7++ZBpDhw7FiRMnxL8L+h6Kym+//YaEhIQ8TnCSk5MREBCAMWPGIDk5GS9evJAtDMeOHcPTp08tjjIbfB1I8a0nTJiAFStWYNasWdi9ezfKlStn0kEq1zoscxTV+6JcJCYmiuvj5OpMPX78OABImmeMMUz7s7en1D/++AOnT59GmzZtcPbsWTg6OmLKlCkm96xatQo3btxA//79Je1A3r17Ny5fvozy5ctj9+7defLJ4cOHxUGlxYsXo2LFipJpG4iLi8Onn35qYjQCukGf6Ohoycvpa9eumTgPW7BgQR6DODU1FefOnZNUV1bMzV+1dFibU3vq1Clx7m5ycjIR6dZ+VKpUyWTucPXq1a3OF4cMaxz79Okjro2TQjczM5OmTZtmMj+9Xbt2VsNw8uRJUqlUdPz4cfFcUdeVtG/fXnyff/31F/n4+Ji8Y+PD3Hx5S3EtzLqDjRs3UuXKlU20uoG+Xu8AAB26SURBVHXrVqRvW5T1Do8ePaLy5csTAAoODrZ6b0F1V6xYQYwxqlatGj148MDk2osXL+iDDz6gcuXKEWOMunTpQmfOnJFE1xrGa3n69etXoP8pqm5kZCRVqFCBvL29xXN37tyhzp07i2tJypYtS1euXCmUbkE4cuRInrRbvnx5mj59Omm12iLFNz/GjRtHZcuWpSdPnojnNBoNXbx4kbp160aVKlWiihUr0uHDhwuVj3Jz7949s+sXQ0ND89w7fPhwk3ucnZ0li68xSUlJVL58ebutcTRgSEcqlYoWLVpUoLVDhdXVaDR06dIl8vf3F/W8vLxo3759lJGRId53/vx5KlGihM3rpAxrHCtXrizWfeYwrHGcOXOmpPE1R6dOncR4eXp6UnBwMOXk5Fi8v7B1UVJSEtWuXdukDqxZsybVrFmTxo4dS3FxcQUKZ2HKqtDQUItrgo3/btSokdm8VVRdA5GRkSQIAvn4+Ji9npqaSq1btya1Wi2p7owZMyzW76VKlaLLly9bjWtRdY3Ztm1bsVrjOG/ePDF9m2t7SVVWdezY0aTMyI/C6Fpap5qWllaocBZW10BSUhKp1Wpq166duP4tOzubNm3aRGq1mlxcXEij0Uime+/ePSpdujS1bNnSbBqKjo4mtVpNKpWKzp07Z1W7KPGNiYmhCRMmkIODg0ke8vPzo7/++ivfdkZRdO/fv0/lypUTtSzZCTdv3rS6Tt6W9Lxp0yYSBKFA9xZEl4ikG3H09vZG1apVAQCdO3fGp59+io4dO5r0ir3xxht49OiRbFPgQkNDERoammf42TBNVfcebOPUqVMoU6YMFi1aBEA3FSg9Pd2k59UaHTp0sEn/+vXr4rTfL774Aj4+Prh48SKaNm2KOXPmYOXKlbL1wO3btw+CIEClUmHIkCF49uwZfH198eDBA+Tk5GD//v12md6o0WjQvXt3JCcno3Xr1hg9erQkzx09ejTWrVuHZ8+ewd3dHYIgwM/PDyEhIShXrhz++OMP/P7770hPT8eBAwcknwefm1WrVonTstq1a4effvpJNq01a9bgzTffxDvvvIMTJ05g3rx5EAQBI0eOxPr166HVarF3716kpaWJI4JS8vbbb4vbuqSnpyMnJwd//fUXli5dikqVKsmyN1pcXBzefvttODs749ChQxAEAaVKlcKePXuwcuVKJCQk4NmzZ/Dz8yu69zE9xnuRGQ7DuihjiCjPvmVycPHiRaSkpODTTz+V5fnmMKzx7Ny5M16+fInJkydLWl4Y0kyJEiXQokUL/Pbbb2jXrh1OnjyJc+fOoUePHiazF1q2bIn4+HhJpqJptVp06NDBqkfrQYMGgYhkH5kDdDMjAGDt2rW4dOkSxo0bJ9nUqJSUFHh6euLRo0cm5+vUqYM6depg165dqF69OtRqNRYvXizrPnsGDh48iPXr1wPQ7Wk5YcIEcc29FNy9exetWrXCpEmTzC6ziYmJQa1atXDv3j2rHtZtpUaNGnjy5AkSExMRFBSEly9fokWLFnjvvfdkGxV8+fIlrly5kmeKsFQQEc6dO4cdO3ZYXXdNRLh27RpcXFwwffp0CIKATz75pEhrxwvK6tWrZdkXG/jvzLzevXtDrVbDzc0NW7duxbx586BSqaBWq9GpUyfZpriXL18eR48eRWpqKtRqNQRBgI+PD95//33k5OTg4MGDUKulW83WsmVLZGZmYtiwYXlGEv/880/Url0bJUqUQFxcHLy8vCTTPnbsGJycnFC7dm0EBwcjJycH77zzjtjOOHz4MNq3by+5l/Fjx47B09MTaWlpYIxh586dZu2EtLQ0cYZbzZo1xXXTUtG5c2e8/fbbUKvVUKvVKFeuHI4cOWKbh21z1qSlIz8LNyIigpo0aSL2/L3//vvUt29f0ZL+8ccfi2Th5qdrPKJo6TD2dllU3Y0bN4ojqBUqVKA7d+7kGx8DJ0+ezGP1F2XEsVq1aiQIAnl4eFBUVBQ1b96cLly4IF5PTk4mZ2dnEgSBmjdvbtaTkqW4WtLNzMyk6dOniz18jDHy8/OjxMTEAse/KLqWCAoKEr/r999/L7nuunXrqFGjRibxbdKkCUVGRhYqnLbGt2HDhiQIAtWpU8fqSIatullZWeTh4UFqtZoePnxIRCR6uTSMCD1//lz0NJqamlooXVsIDQ3N11tlUXWDgoKoXr16dPXqVXJzc6MhQ4bQ3bt3Cxy2gupaGnHM7TH14cOH1KZNmzz35fbQKMV7/uyzz0gQBJMZEFLF1xyxsbHUsWNHYozRu+++W6iwFkQ3JyeHvvnmG5M8O2jQoHxHNOfMmSPZiKMgCLRgwQLRK+/UqVPp8OHD5OHhQZcuXaLatWuTIAi0fft2m+ObHxMmTBBnT0RHR+d7f2HqotjYWBOv4aVLl6ZJkyaJ1+Pj42nAgAHi9YCAAEpKSipUXM3pnj17lqZMmUJTpkyhzz//3Ky3RSKi+vXrkyAI1KtXr0KXVZbKyBkzZhAAk7rWmJEjRxKAfGeFFKVOePLkCd24cYNu3LhB//zzj3heq9XSlClTxLR39uxZSXUNJCQkiKO6Y8eOlWQEzlh337599NZbbxFjjGrWrEmTJk2i6OhoOn36NF27do3i4uLo8ePHdObMGZP8/d577xVat6BkZGRQ1apVTd63LfE1x+nTp8nPz4/8/PwoLCwsz/VVq1aRi4sLCYJA/v7+VtsAtsQ3OzubYmJiKDIyUnzHFSpUyON51FZdZ2dnYoxR3bp16a+//qKoqCg6f/48rVq1ikqVKiWO8J4+fZrS09OtjgAWVDcjI4PatWtnMsI4Z86cInmuLYzu48ePTUYa/fz8zD4vOzubxo4dS4IgUPfu3S2+c1vTs0ajoQcPHtCuXbuoWbNmpFKp8syqK6guEUm/HQeRrmK5f/8+EZHoOrpbt24Fcgdb2BdUEKPR0hYchdU1TC1wdXWlq1ev5vtMY9q0aUMqlcrkXGENx+joaCpdujQJgkC7du2iVatW0cGDB8XrOTk5NHHiRBIEgUqXLi2JS/DMzEz66aefxAzAGKN//etfkrn2LazhmJWVRe7u7gSAWrdubdP0Amu6ycnJVLVqVZOKqkuXLnTixAmKiooqUOVpS3xv374tum5u1KhRvvfbonv37l1ijNH48eOJSDdld/To0eKUGYPRyBij//u//yu0ri0YDBy5DEfDt42NjS102KQ0HJ88eWLWaFSr1bRhwwZJ4muMn58fubm5WWxY2xJfc8yfP1981+a26rFVNyoqyiSvlitXLt+tXK5cuWKy9U5RdIl0ZdKVK1foypUrtHLlSpOy0tzvffr0sTm++REbG0u9evUilUpFvr6++TaUbDEczU13TklJoS+++EK8x8PDw2xHo1SdicY0bNiQ1Go1DRkyxGI5XRjdmJgYqlixIg0fPtxsfbN161ZSqVQ0atSofN+z1PE1bHOTX7qSynDcuXNnocJXEF1PT09ijBEAYoyRWq2matWqictCypUrR2XLlhXzKQDy9PS02pFrax6KiIiwapjKpZubq1evitPBR40aZbKsQg5dQzk9aNCgAt1fGN2wsDAqU6aMSTltfLi5uVHz5s3FNGBpSUxhdHNP8/7mm2/EPPz06VMKCgqiESNG0LvvvksrVqwQtyHJysoyO7W/oLo7d+4UNa1Nd75165Z4n7VOJynT1ZMnT0ilUpG/vz9lZmZavdeuhqOB06dPi1b3ihUrCvQ/hXlBt2/fFo1D40JTrhFHg+HYq1evAsXFwNq1a8V9Ho0pTGWt1Wrpww8/FPdhS0pKopycHNGA02g0FBERISZCa+v+ClqJREdHi/vpGY5Zs2ZZXStjDSkqTQ8PD/GbHjt2TDbd9PR0cnFxIcYYVaxYkRo2bGhSyH300Uf5Go9Fje+NGzeoatWqYmWd31pKW3WHDx9OjDG6efMmERG1bt1aNBrPnj0rVuwrV6602jCSutK8dOmSWGlKaThmZ2fThQsXxMaI8WhJYbDVcGzYsCG5urqSq6srVaxY0eIej7nfuRTv2dC7K0d8c3Po0CEqVaoUlS5d2qyBJoVutWrVTPJnqVKlyMvLi7y8vOj999+n48eP0/HjxykkJEQ8b7y+8e+//5YkvsnJyeTl5UWCIBAAsdw0/t0eI44G/Pz8iDFGQUFBVu+T2nAk0o08urm5iXnY3EiznIbjlClTLN5TGN3AwEACQEePHs1zbe/evaKPA1vW69oS3+3bt/9PG46GGUzHjx+nOXPm0LBhwywaF4Zjzpw5RdItKCNGjChw+0JKXUs0b96cBEGgkSNH2rTe3hqZmZmisX7r1q0C/U9hdVesWCGOPJo7BgwYQN9++22+o50FnYXy6aefmrRfvb29aebMmTRz5kyL64YNx5gxY/K08Qqim5qaKn4vQRBo3rx5ecL/8uVLWrt2LZUoUYIEQaD69etb3ZNc6nRlGISaP3++1fssGY6SbseRm/bt24OI8MUXX2DUqFGSP3/69OkAgJCQEIwdOzbP9T59+ojrGxs2bKizlCWgQoUKhbp/xIgRYIzZNFc+OzsbP//8MwDdWjxjV+A5OTn48ssvsXjxYpQsWVLcQsAWDFtsPHnyxGTu961bt8Q1Qe7u7qLX00aNGuHWrVuoUqUKypQpgx49etikb4nr168D0O2bJaeH0ffffx9PnjxB+fLlcejQIdSvXx/Xr19HcHAwzp49i59++gnvvvsuevfubXVdU2G5d+8eevbsicTERAC69Q+NGzeW7PnmuHTpEpycnODq6op169bh/PnzICIEBgYiODgYjo6OOHDgAN555x1J1gE8ePAAtWrVsrqG4cmTJwgODpYszxqzc+dO9O/fH40bN8aNGzfg5eUluUZBuHfvnvi7Vqu1uAbN1vWVlrDHvmwpKSkYPXo0srKyMGvWLMycOVMWnadPn5r8nZWVJa4Fv3DhgsX9sZo1awY/Pz8Tz6u24OzsjIiICCxatAhEhI8//hjVqlUDAEybNg0LFy6UJU2b4/bt2zhz5gyA/+5lKQVubm746aefMHz4cGRnZyMpKQnr16/H0KFDTe6rXLkytm3bBh8fH+Tk5ODSpUs2a8fHx1vd1+3XX39FVFSUzTrG7NmzB15eXmjZsqXJ+YSEBIwdOxYeHh5YsWKF3bcvMjBixAhZn1+uXDl89dVXmD17tmwa7dq1Q9u2bdGhQwekpqaic+fOGDZsGDQaDQDdAIeh7pk1a5akW0SYY/LkyTh69Kgk26pIwZkzZzBu3DisXr0aTZs2lWVt+pYtW/DixQt07dpVNk+yo0ePRvfu3XHnzh0cOHAAixYtQoUKFTBv3jwMGzYMarVasrWGn3zySR7P4WfPni2wj4YVK1agU6dO6Nu3b6F0586dK26l07dvX0yePNnkelpaGry9vcW9IytXroxNmzbJtq2MOWrWrAlfX1/ExsYW6f9lNRwNFWSLFi0kX3gaGhoqGoXGRmNAQAAAndG4fft2BAQESL6HY4sWLQp0X2ZmJv7zn/+If4eGhkqib5zAIiMjMXHiRBw4cABqtRqHDx+Gt7e3zRotWrQQt6cwbugY9jYyh3HhHhwcbNaYtwXDBuIAMHXqVMkXERvIzMwUF9wfPHhQ3BLC29sb3t7eePz4Md588030798fw4YNk2xbg3v37qF+/fomBsSuXbskeXZ+MMaQmpqKQ4cOgYjQqlUr3L59GwEBAQgLC5Nsg+3k5GR4eHjgzJkzZp2EaLVaREVFoW3btkhISECVKlWwZs0aq3umFYagoCBMnDgRgYGB+PLLL9GpUyfUrVtXkmfLgZOTk2zP/uijj2R7toFRo0bh4cOH4sbochmObdu2zbOBtSUaNmwIf39/eHt7o2vXrpI3+B0dHfH111/nOT948GAsWrRI8rowNy9evMCkSZOwc+dOZGRkgIhQq1YtSTUGDx6MnJwcfPzxx0hPT8fw4cPRqlUrODs7w83NDRqNBi9evEDbtm3h6uqK2NhYm43Xc+fOYcSIERb3U01PT0doaCiys7Nt0jHm0aNHuHXrFhYuXJhnD0xvb288e/YMq1atgoeHh2SahWH9+vWi4zCpN2o3kJqaitmzZ6Nq1aqSpyMD3333Hc6fP4+yZcsiLS0Nf/75p8l1xhjCw8Ph5uYGd3d3SZ22mCM6Ohq+vr6yahQGBwcHrFy5Ek+fPsW1a9dk0Zg3bx4YY/joo48k3Sw+N7Vq1YKTk5PYbtqyZYvVPbOLyqpVq6w6BDO0k0+fPm1yftGiRejcuTMA2Jyvf/zxR5N2akpKCho1aiSWhYIgYOfOnXk6pexFUR1LyZb77t+/D0BnyPn7+8slgz59+oi/Gypk4xHI7du3S1ZRG0YFDh06hEmTJlm8LyEhAcuXLxcbSVqtFrt27ZLsPezZswc+Pj44c+YMpk2bhuzsbJQpUwbjx4+XxGicNWsWEhMTwRgzMQYBU+Mw9zVA9w1cXFxk8aRnGIVr2bKlrHui7d69W9Qy10Bxc3PDrl270KVLF2zfvh1ffvkl3njjDZt1V6xYYbLJsjmvm3LQsmVLXL58GbVr1xbPRUVFYfny5fjwww8lHVHNyclBVlYWRo4cCU9PT4wYMcKkQTZ37lyT0aGvv/4aPXv2lET79u3b2LhxI7788kvMmDEDAPD48WPRG7Rc1KpVC/Pnz8cXX3xR6P+tWbOmDCHSIfe+gjdu3MChQ4cA6NK2oSNKDg4fPowTJ07g2LFj+P3333Hx4kUAujJq6NChqFu3LkaMGIEKFSpArVbL5ilRKc6dO4e4uDj89ttvOHToEP7++2/x2rRp0zBs2DDJNYcOHQq1Wo0dO3Zg165dYkdQv379kJKSgkuXLsHPz8/ku2dlZRW5PDl37hxu3LiBNWvWYO7cuSbXwsPDMXXqVLGnv3fv3vjhhx+KGLP/Mn/+fBCRSQNUo9EgJCQEsbGxOHv2rF285MbFxaFatWpifavRaHD27FnRc2zt2rVN9uyUGkEQ0KVLlwJ3mheGli1b4sKFC/jzzz/ztCeICOPHj8fixYslHY3Kj23btuUZKSoOeHh4YP78+Vi5cqXkz46JiUGLFi0kq2+tERkZKe7t7uPjI7seANStWxddunQBAPTo0QNdu3YFoGtzLF68WPRK7OXlJVlH0NWrV/H8+XPk5OTgu+++w8mTJ0060CZPnox27dpJolUYUlNTcf/+/SLPVpDFcMzJyUFQUBAqVqyI5cuXy947FBoainHjxol/5x7pCgkJMbleVDp27IiTJ0/iyJEjCA4OxsCBA1G5cmXxemxsLKKjozFy5EjcuXNHLOSeP39u8xYZgiDA19cXR48exalTp0y29WjVqhWWLVsmWaGekpJi8rezszPatWuH+vXrw83NDZUqVUL37t0t/r+Tk1Oe3lkpWLVqFQDdVEM5KVOmjGg0h4eHi6PYBjIzM3Hr1i0AQKlSpSQbjcvNsmXLZHlubkJCQlC1alXMmzcPANC9e3f88ssvsmy+yxiDIAg4f/48zp8/j7CwMIv3NmvWDB988IFk2jt37sSZM2fQqlUrlC5dGllZWahatSqWLFmC4OBgyXRy4+DggI4dO6JGjRp5tjCwRP369dG+fXssXrxY8vBoNBr4+vrKPkqyZs0aJCcno1GjRnj77bdx/fp1uLq6yqJVsmRJdOrUCZ06dcL06dNNOq5KlCghaw+6vdFoNOJ2UGfOnMHevXuRk5NjMjOkVKlS+O677+Dv7486derI1uAeNGgQBgwYgMjISPj6+uLZs2fYsmWLqGdoHBqwZUuQKlWqoESJEvj+++/h4uKCDz/8EE5OTkhPT0fXrl3FTrcOHTpgw4YNNsXLgKGR16NHD7x48QKXL1/G9OnTcfz4cVy7dk32PKTRaLBgwQKsXr0ab7zxhjhKPmPGDJPRgv3798s21U2tVsPV1RVpaWnIzMyUvNPlq6++wpkzZ7B8+XKx7fHmm29iyZIlaNiwoUmH5qvI0qVL4efnl28HxJkzZzB37ly89957soQjPT0dPXv2lHWWC6AzWgxtmw4dOsi2FGPy5MlYsmQJAODbb7/F5MmTzbbDZ8+ejc8//xwajQZLliyxubN23Lhx2Lx5M6Kjo/Huu+9avK9Jkyb46quvbNIqClqtFj///DMeP35c9Jlc5hY+WjoKuhhz69atJAgCfffddwW6P7/FmOZ0Q0JC8jjA6dOnj1knOIZ7bdVNTU2lbt26iU4BGjZsSN27d6cePXpQjx49qE6dOqIDHZVKRd27dxe3NzBHYb2qpqWl0ZgxY0wW8EZERBTKM6K1uBp0r1+/ToIg0FtvvUULFy4s1FYQtuhaIzk5mQRBoCFDhlBWVpbsurVq1RI3z963bx9FR0fT7Nmzafbs2eTu7i5uYm5tkX5hddetW2eykfXAgQMLFU9b4isFBc27e/bsobp161pdmD579mx6/vy5pLoRERHEGKPhw4cTkc4BUq1atcw6vZBS18CZM2eofPnyZh3gGH93tVpN06dPl0w3NxqNhgDQixcvCvw/RdE1OKzZtGkTvXz5koKDg2nmzJmF0iyKrlRIrXvp0iVijNG2bdts0n3w4EG+zkPCw8MLFbaibA2Vm/T0dHrx4gUlJSXRDz/8QGPHjhXrwh9++IHWr19f4Lha0jU4vhEEgapWrUqhoaHk4+Njkn/ye7+F0e3bty8BIGdnZ6pcubLo5KhBgwYFfi+2xDclJUX0rm3p+OCDD2T35rpt2zbZnOPIga15d/369eTo6Gg2zUqtm5ycTKNGjaJBgwbR3r1781w/ffo0rVq1SnQ0uWXLFkl0jTl48CA5ODjQvXv3Cvw/RdU1eDN3dHSkR48eFUqvMLparZYmT55MgiDQ2rVri6RTFF0iomvXrlnNsytWrBAdEEqpa0xmZmYer6nPnz+noUOHkkqloiNHjhRJl0gG5zifffYZli5dCkEQxA1O5WDs2LEFXkNXmHutUbZsWezfvx8AcPnyZcydOxc7duwQe1c3bNiA5s2bo0GDBjZrmcPR0RGhoaGSrZW0ROPGje2yaXNhGDx4sNhTYg8MU7127NiB2NhYhIWFwd3dHTt27MCyZcvEKQ9S8uGHH8o63ai40KtXL/Tq1cvuuj4+Prh06RJmzpyJnj17okuXLiZT+uSmdevWSEpKspueJcLDwzFs2DCbZ0HkR8WKFfH06VO8//77YIzB2dm5WMRfKRhj4mELdevWtYtjo8JivFZ0ypQpAHSzGaTk1q1buH//Pv7++28EBgZiwoQJAHRT38wtK7CVrVu3Sv7MwuDk5ITw8HBcuXIFhw8fxrp16wDoHAPK6awmNwEBAcWuTSAnH3zwgaSzXazh7OwsTj1NT09H9erVERcXB8YYPD09wRjD0KFDcePGDVmWLixbtgzjxo3DnTt3ZMlDuRk/fjzGjx8vuw5jDIsWLRJnZ9gTDw8PRfPL8+fPkZSUJPokSUlJEWfGTZgwAWFhYTaN9EpuOBqmGvzrX/+S+tHFCk9PT8UrFY78GNbQjhw5EgDw/fffKxkcjo00a9bMbg6HiiuNGze2yxSZQ4cOoXXr1nj8+DHeeecdbNy4UXbN4kydOnUQFBSkdDD+53F3d4e7uzvOnTundFDsgoeHBzw8PDB48GDJHLFxiieOjo5F9nRZVMLDwwGgWDuJ4xQOJycnODk5Ydq0abI8X5bFhyVLlsS3334rx6M5rykzZ84U3etzOJyiU6NGDbvouLm5FXhN5+uAs7Oz5J6mORwOxxa2bdumdBA4/2MUfaW6BdasWYP09HS7eBrjvD60atUK//zzj9LB4HA4HA6Hw+FwXkuYbv1jAW9mLBXAHfmCAwCoTUQmO/y+wrrmNOMBpAFI4Lpc9xXQfVXzLtfluq+SbnEqM7gu15VD91XNu1yX69pNFyj8VNU7ROQlUYC4rhmIqApj7DzX5bqvgi5eo7zLdbnuq6T7upVVXPfV1sVrlHe5LteVE8mnqnI4HA6Hw+FwOBwO59WCG44cDofD4XA4HA6Hw7FKYQ3HlbKEgutyXa77quq+TnHlulyX63Jdrls8dV+nuHJdrisbhXKOw+FwOBwOh8PhcDic1w8+VZXD4XA4HA6Hw+FwOFbhhiOHw+FwOBwOh8PhcKzCDUcOh8PhcDgcDofD4ViFG44cDofD4XA4HA6Hw7EKNxw5HA6Hw+FwOBwOh2OV/wd2Tmpd4sKDkQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x432 with 30 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from utils import load_data\n",
    "\n",
    "# note we also shrink the intensity values (X) from 0-255 to 0-1. This helps the neural network converge faster.\n",
    "X_train = load_data('./data/mnist/train-images.gz', False) / 255.0\n",
    "y_train = load_data('./data/mnist/train-labels.gz', True).reshape(-1)\n",
    "\n",
    "X_test = load_data('./data/mnist/test-images.gz', False) / 255.0\n",
    "y_test = load_data('./data/mnist/test-labels.gz', True).reshape(-1)\n",
    "\n",
    "count = 0\n",
    "sample_size = 30\n",
    "plt.figure(figsize = (16, 6))\n",
    "for i in np.random.permutation(X_train.shape[0])[:sample_size]:\n",
    "    count = count + 1\n",
    "    plt.subplot(1, sample_size, count)\n",
    "    plt.axhline('')\n",
    "    plt.axvline('')\n",
    "    plt.text(x = 10, y = -10, s = y_train[i], fontsize = 18)\n",
    "    plt.imshow(X_train[i].reshape(28, 28), cmap = plt.cm.Greys)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Dataset for Files\n",
    "A Dataset can reference single or multiple files in your datastores or public urls. The files can be of any format. Dataset provides you with the ability to download or mount the files to your compute. By creating a dataset, you create a reference to the data source location. If you applied any subsetting transformations to the dataset, they will be stored in the dataset as well. The data remains in its existing location, so no extra storage cost is incurred. [Learn More](https://aka.ms/azureml/howto/createdatasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.dataset import Dataset\n",
    "web_paths = ['http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz',\n",
    "             'http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz',\n",
    "             'http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz',\n",
    "             'http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz'\n",
    "            ]\n",
    "dataset = Dataset.File.from_files(path = web_paths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may want to regiester datasets using the register() method to your workspace so they can be shared with others, reused across various experiments, and referred to by name in your training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['/http/yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz',\n",
       "       '/http/yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz',\n",
       "       '/http/yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz',\n",
       "       '/http/yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = dataset.register(workspace = ws,\n",
    "                           name = 'mnist dataset',\n",
    "                           description='training and test dataset',\n",
    "                           create_new_version=True)\n",
    "# list the files referenced by dataset\n",
    "dataset.to_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create or Attach existing AmlCompute\n",
    "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this tutorial, you create `AmlCompute` as your training compute resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we could not find the cluster with the given name, then we will create a new cluster here. We will create an `AmlCompute` cluster of `STANDARD_NC6` GPU VMs. This process is broken down into 3 steps:\n",
    "1. create the configuration (this step is local and only takes a second)\n",
    "2. create the cluster (this step will take about **20 seconds**)\n",
    "3. provision the VMs to bring the cluster to the initial size (of 1 in this case). This step will take about **3-5 minutes** and is providing only sparse output in the process. Please make sure to wait until the call returns before moving to the next cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating a new compute target...\n",
      "Creating\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "Minimum number of nodes requested have been provisioned\n",
      "{'currentNodeCount': 0, 'targetNodeCount': 0, 'nodeStateCounts': {'preparingNodeCount': 0, 'runningNodeCount': 0, 'idleNodeCount': 0, 'unusableNodeCount': 0, 'leavingNodeCount': 0, 'preemptedNodeCount': 0}, 'allocationState': 'Steady', 'allocationStateTransitionTime': '2019-11-13T09:18:49.696000+00:00', 'errors': None, 'creationTime': '2019-11-13T09:18:46.043348+00:00', 'modifiedTime': '2019-11-13T09:19:01.810871+00:00', 'provisioningState': 'Succeeded', 'provisioningStateTransitionTime': None, 'scaleSettings': {'minNodeCount': 0, 'maxNodeCount': 4, 'nodeIdleTimeBeforeScaleDown': 'PT120S'}, 'vmPriority': 'Dedicated', 'vmSize': 'STANDARD_NC6'}\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# choose a name for your cluster\n",
    "cluster_name = \"gpu-clustertf\"\n",
    "\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print('Found existing compute target')\n",
    "except ComputeTargetException:\n",
    "    print('Creating a new compute target...')\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_NC6', \n",
    "                                                           max_nodes=4)\n",
    "\n",
    "    # create the cluster\n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "\n",
    "    # can poll for a minimum number of nodes and for a specific timeout. \n",
    "    # if no min node count is provided it uses the scale settings for the cluster\n",
    "    compute_target.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
    "\n",
    "# use get_status() to get a detailed status for the current cluster. \n",
    "print(compute_target.get_status().serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have created the compute target, let's see what the workspace's `compute_targets` property returns. You should now see one entry named 'gpu-cluster' of type `AmlCompute`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aks-aml-visual AKS Succeeded\n",
      "aks-cluster01 AKS Succeeded\n",
      "aksdevtest AKS Succeeded\n",
      "vmds13v2 VirtualMachine Succeeded\n",
      "vmstandardnc12 VirtualMachine Succeeded\n",
      "StandardDS5V2 AmlCompute Succeeded\n",
      "automlcl AmlCompute Succeeded\n",
      "aml-compute AmlCompute Succeeded\n",
      "cpu-cluster AmlCompute Succeeded\n",
      "gpu-cluster AmlCompute Succeeded\n",
      "gpu-cluster2 AmlCompute Succeeded\n",
      "gpu-cluster3 AmlCompute Succeeded\n",
      "gpu-clustertf AmlCompute Succeeded\n"
     ]
    }
   ],
   "source": [
    "compute_targets = ws.compute_targets\n",
    "for name, ct in compute_targets.items():\n",
    "    print(name, ct.type, ct.provisioning_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy the training files into the script folder\n",
    "The TensorFlow training script is already created for you. You can simply copy it into the script folder, together with the utility library used to load compressed data file into numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./tensorflow-mnist/utils.py'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import shutil\n",
    "\n",
    "# the training logic is in the tf_mnist.py file.\n",
    "shutil.copy('./tf_mnist.py', script_folder)\n",
    "\n",
    "# the utils.py just helps loading data from the downloaded MNIST dataset into numpy arrays.\n",
    "shutil.copy('./utils.py', script_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "2039d2d5-aca6-4f25-a12f-df9ae6529cae"
    }
   },
   "source": [
    "## Construct neural network in TensorFlow\n",
    "In the training script `tf_mnist.py`, it creates a very simple DNN (deep neural network), with just 2 hidden layers. The input layer has 28 * 28 = 784 neurons, each representing a pixel in an image. The first hidden layer has 300 neurons, and the second hidden layer has 100 neurons. The output layer has 10 neurons, each representing a targeted label from 0 to 9.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Azure ML concepts  \n",
    "Please note the following three things in the code below:\n",
    "1. The script accepts arguments using the argparse package. In this case there is one argument `--data_folder` which specifies the file system folder in which the script can find the MNIST data\n",
    "```\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument('--data_folder')\n",
    "```\n",
    "2. The script is accessing the Azure ML `Run` object by executing `run = Run.get_context()`. Further down the script is using the `run` to report the training accuracy and the validation accuracy as training progresses.\n",
    "```\n",
    "    run.log('training_acc', np.float(acc_train))\n",
    "    run.log('validation_acc', np.float(acc_val))\n",
    "```\n",
    "3. When running the script on Azure ML, you can write files out to a folder `./outputs` that is relative to the root directory. This folder is specially tracked by Azure ML in the sense that any files written to that folder during script execution on the remote target will be picked up by Run History; these files (known as artifacts) will be available as part of the run history record."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell will print out the training code for you to inspect it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Copyright (c) Microsoft Corporation. All rights reserved.\n",
      "# Licensed under the MIT License.\n",
      "\n",
      "import numpy as np\n",
      "import argparse\n",
      "import os\n",
      "import tensorflow as tf\n",
      "import glob\n",
      "\n",
      "from azureml.core import Run\n",
      "from utils import load_data\n",
      "\n",
      "print(\"TensorFlow version:\", tf.__version__)\n",
      "\n",
      "parser = argparse.ArgumentParser()\n",
      "parser.add_argument('--data-folder', type=str, dest='data_folder', help='data folder mounting point')\n",
      "parser.add_argument('--batch-size', type=int, dest='batch_size', default=50, help='mini batch size for training')\n",
      "parser.add_argument('--first-layer-neurons', type=int, dest='n_hidden_1', default=100,\n",
      "                    help='# of neurons in the first layer')\n",
      "parser.add_argument('--second-layer-neurons', type=int, dest='n_hidden_2', default=100,\n",
      "                    help='# of neurons in the second layer')\n",
      "parser.add_argument('--learning-rate', type=float, dest='learning_rate', default=0.01, help='learning rate')\n",
      "args = parser.parse_args()\n",
      "\n",
      "data_folder = args.data_folder\n",
      "print('Data folder:', data_folder)\n",
      "\n",
      "# load train and test set into numpy arrays\n",
      "# note we scale the pixel intensity values to 0-1 (by dividing it with 255.0) so the model can converge faster.\n",
      "X_train = load_data(glob.glob(os.path.join(data_folder, '**/train-images-idx3-ubyte.gz'),\n",
      "                              recursive=True)[0], False) / 255.0\n",
      "X_test = load_data(glob.glob(os.path.join(data_folder, '**/t10k-images-idx3-ubyte.gz'),\n",
      "                             recursive=True)[0], False) / 255.0\n",
      "y_train = load_data(glob.glob(os.path.join(data_folder, '**/train-labels-idx1-ubyte.gz'),\n",
      "                              recursive=True)[0], True).reshape(-1)\n",
      "y_test = load_data(glob.glob(os.path.join(data_folder, '**/t10k-labels-idx1-ubyte.gz'),\n",
      "                             recursive=True)[0], True).reshape(-1)\n",
      "\n",
      "print(X_train.shape, y_train.shape, X_test.shape, y_test.shape, sep='\\n')\n",
      "\n",
      "training_set_size = X_train.shape[0]\n",
      "\n",
      "n_inputs = 28 * 28\n",
      "n_h1 = args.n_hidden_1\n",
      "n_h2 = args.n_hidden_2\n",
      "n_outputs = 10\n",
      "learning_rate = args.learning_rate\n",
      "n_epochs = 20\n",
      "batch_size = args.batch_size\n",
      "\n",
      "with tf.name_scope('network'):\n",
      "    # construct the DNN\n",
      "    X = tf.placeholder(tf.float32, shape=(None, n_inputs), name='X')\n",
      "    y = tf.placeholder(tf.int64, shape=(None), name='y')\n",
      "    h1 = tf.layers.dense(X, n_h1, activation=tf.nn.relu, name='h1')\n",
      "    h2 = tf.layers.dense(h1, n_h2, activation=tf.nn.relu, name='h2')\n",
      "    output = tf.layers.dense(h2, n_outputs, name='output')\n",
      "\n",
      "with tf.name_scope('train'):\n",
      "    cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=output)\n",
      "    loss = tf.reduce_mean(cross_entropy, name='loss')\n",
      "    optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
      "    train_op = optimizer.minimize(loss)\n",
      "\n",
      "with tf.name_scope('eval'):\n",
      "    correct = tf.nn.in_top_k(output, y, 1)\n",
      "    acc_op = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
      "\n",
      "init = tf.global_variables_initializer()\n",
      "saver = tf.train.Saver()\n",
      "\n",
      "# start an Azure ML run\n",
      "run = Run.get_context()\n",
      "\n",
      "with tf.Session() as sess:\n",
      "    init.run()\n",
      "    for epoch in range(n_epochs):\n",
      "\n",
      "        # randomly shuffle training set\n",
      "        indices = np.random.permutation(training_set_size)\n",
      "        X_train = X_train[indices]\n",
      "        y_train = y_train[indices]\n",
      "\n",
      "        # batch index\n",
      "        b_start = 0\n",
      "        b_end = b_start + batch_size\n",
      "        for _ in range(training_set_size // batch_size):\n",
      "            # get a batch\n",
      "            X_batch, y_batch = X_train[b_start: b_end], y_train[b_start: b_end]\n",
      "\n",
      "            # update batch index for the next batch\n",
      "            b_start = b_start + batch_size\n",
      "            b_end = min(b_start + batch_size, training_set_size)\n",
      "\n",
      "            # train\n",
      "            sess.run(train_op, feed_dict={X: X_batch, y: y_batch})\n",
      "        # evaluate training set\n",
      "        acc_train = acc_op.eval(feed_dict={X: X_batch, y: y_batch})\n",
      "        # evaluate validation set\n",
      "        acc_val = acc_op.eval(feed_dict={X: X_test, y: y_test})\n",
      "\n",
      "        # log accuracies\n",
      "        run.log('training_acc', np.float(acc_train))\n",
      "        run.log('validation_acc', np.float(acc_val))\n",
      "        print(epoch, '-- Training accuracy:', acc_train, '\\b Validation accuracy:', acc_val)\n",
      "        y_hat = np.argmax(output.eval(feed_dict={X: X_test}), axis=1)\n",
      "\n",
      "    run.log('final_acc', np.float(acc_val))\n",
      "\n",
      "    os.makedirs('./outputs/model', exist_ok=True)\n",
      "    # files saved in the \"./outputs\" folder are automatically uploaded into run history\n",
      "    saver.save(sess, './outputs/model/mnist-tf.model')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(os.path.join(script_folder, './tf_mnist.py'), 'r') as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create TensorFlow estimator\n",
    "Next, we construct an `azureml.train.dnn.TensorFlow` estimator object, use the Batch AI cluster as compute target, and pass the mount-point of the datastore to the training code as a parameter.\n",
    "\n",
    "The TensorFlow estimator is providing a simple way of launching a TensorFlow training job on a compute target. It will automatically provide a docker image that has TensorFlow installed -- if additional pip or conda packages are required, their names can be passed in via the `pip_packages` and `conda_packages` arguments and they will be included in the resulting docker.\n",
    "\n",
    "The TensorFlow estimator also takes a `framework_version` parameter -- if no version is provided, the estimator will default to the latest version supported by AzureML. Use `TensorFlow.get_supported_versions()` to get a list of all versions supported by your current SDK version or see the [SDK documentation](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.dnn?view=azure-ml-py) for the versions supported in the most current release."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "dnn-tensorflow-remarks-sample"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - framework_version is not specified, defaulting to version 1.13.\n"
     ]
    }
   ],
   "source": [
    "from azureml.train.dnn import TensorFlow\n",
    "\n",
    "script_params = {\n",
    "    '--data-folder': dataset.as_named_input('mnist').as_mount(),\n",
    "    '--batch-size': 50,\n",
    "    '--first-layer-neurons': 300,\n",
    "    '--second-layer-neurons': 100,\n",
    "    '--learning-rate': 0.01\n",
    "}\n",
    "\n",
    "est = TensorFlow(source_directory=script_folder,\n",
    "                 script_params=script_params,\n",
    "                 compute_target=compute_target,\n",
    "                 entry_script='tf_mnist.py',\n",
    "                 use_gpu=True,\n",
    "                 pip_packages=['azureml-dataprep[pandas,fuse]'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit job to run\n",
    "Submit the estimator to an Azure ML experiment to kick off the execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = exp.submit(est)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor the Run <a class=\"anchor\" id=\"monitor-run\"></a>\n",
    "As the Run is executed, it will go through the following stages:\n",
    "1. Preparing: A docker image is created matching the Python environment specified by the TensorFlow estimator and it will be uploaded to the workspace's Azure Container Registry. This step will only happen once for each Python environment -- the container will then be cached for subsequent runs. Creating and uploading the image takes about **5 minutes**. While the job is preparing, logs are streamed to the run history and can be viewed to monitor the progress of the image creation.\n",
    "\n",
    "2. Scaling: If the compute needs to be scaled up (i.e. the Batch AI cluster requires more nodes to execute the run than currently available), the cluster will attempt to scale up in order to make the required amount of nodes available. Scaling typically takes about **5 minutes**.\n",
    "\n",
    "3. Running: All scripts in the script folder are uploaded to the compute target, data stores are mounted/copied and the `entry_script` is executed. While the job is running, stdout and the `./logs` folder are streamed to the run history and can be viewed to monitor the progress of the run.\n",
    "\n",
    "4. Post-Processing: The `./outputs` folder of the run is copied over to the run history\n",
    "\n",
    "There are multiple ways to check the progress of a running job. We can use a Jupyter notebook widget. \n",
    "\n",
    "**Note: The widget will automatically update ever 10-15 seconds, always showing you the most up-to-date information about the run**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4ec9a4c3a0d45b5bbc56cae5e3944de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': True, 'log_level': 'INFO', 's…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlserviceresourcegroup/workspaces/azuremlservice\", \"run_id\": \"tf-mnist_1573636755_646589f7\", \"run_properties\": {\"run_id\": \"tf-mnist_1573636755_646589f7\", \"created_utc\": \"2019-11-13T09:19:17.664176Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"batchai\", \"ContentSnapshotId\": \"2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850\", \"AzureML.DerivedImageName\": \"azureml/azureml_75651667c3311d0b90ca821953173db3\", \"ProcessInfoFile\": \"azureml-logs/process_info.json\", \"ProcessStatusFile\": \"azureml-logs/process_status.json\"}, \"tags\": {\"_aml_system_ComputeTargetStatus\": \"{\\\"AllocationState\\\":\\\"steady\\\",\\\"PreparingNodeCount\\\":1,\\\"RunningNodeCount\\\":0,\\\"CurrentNodeCount\\\":1}\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": \"2019-11-13T09:27:34.11495Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/55_azureml-execution-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/55_azureml-execution-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt?sv=2019-02-02&sr=b&sig=67v5FY6T4ZWeivxKFHHGRSkMZP1DPdxarahslGnbvNQ%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"azureml-logs/65_job_prep-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/65_job_prep-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt?sv=2019-02-02&sr=b&sig=YpE0g79U%2FHxIYEOkSScPks5NW72zJ5YEVT5bShP9yoM%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"azureml-logs/70_driver_log.txt\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=G4BwR8lmF08S%2Bgvg43I33b7odNM7slTSAWUo3s%2F6Jb8%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"azureml-logs/75_job_post-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/75_job_post-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt?sv=2019-02-02&sr=b&sig=RjjhYs9yFZgfwSnLvA4pGqPUwQp7x25h9BI3CZvHVWE%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"azureml-logs/process_info.json\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=savSZcmZCzSXXViOOr1djEZ7CnO44PwEWknI211R%2Ft4%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"azureml-logs/process_status.json\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=gC3A%2Fh4snFd5R57c8tGcQtM1sJsGJ2RvuEdDcEc9wpQ%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"logs/azureml/140_azureml.log\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/logs/azureml/140_azureml.log?sv=2019-02-02&sr=b&sig=B2BYX%2Fqp2g2F9hbShicjzGOuVr1FnNFqKyg6cBT6bTg%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\", \"logs/azureml/azureml.log\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/logs/azureml/azureml.log?sv=2019-02-02&sr=b&sig=eL4FuGVteqqsgtnOea657h%2FQXh0KU%2BGGmoYETMJPjnQ%3D&st=2019-11-13T09%3A17%3A39Z&se=2019-11-13T17%3A27%3A39Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/process_info.json\", \"azureml-logs/process_status.json\", \"logs/azureml/azureml.log\"], [\"azureml-logs/55_azureml-execution-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt\"], [\"azureml-logs/65_job_prep-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt\"], [\"azureml-logs/70_driver_log.txt\"], [\"azureml-logs/75_job_post-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt\"], [\"logs/azureml/140_azureml.log\"]], \"run_duration\": \"0:08:16\"}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [{\"name\": \"training_acc\", \"run_id\": \"tf-mnist_1573636755_646589f7\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], \"series\": [{\"data\": [0.8399999737739563, 0.9599999785423279, 0.9399999976158142, 0.9399999976158142, 0.9599999785423279, 0.9599999785423279, 1, 0.9800000190734863, 1, 0.9800000190734863, 0.9800000190734863, 0.9800000190734863, 0.9599999785423279, 1, 0.9599999785423279, 0.9399999976158142, 1, 0.9599999785423279, 0.9399999976158142, 0.9599999785423279]}]}, {\"name\": \"validation_acc\", \"run_id\": \"tf-mnist_1573636755_646589f7\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], \"series\": [{\"data\": [0.9050999879837036, 0.9207000136375427, 0.930899977684021, 0.9397000074386597, 0.9455999732017517, 0.9480000138282776, 0.951200008392334, 0.9556999802589417, 0.9580000042915344, 0.9602000117301941, 0.9605000019073486, 0.9643999934196472, 0.9650999903678894, 0.9675999879837036, 0.9682999849319458, 0.9674000144004822, 0.9700000286102295, 0.970300018787384, 0.97079998254776, 0.9725000262260437]}]}, {\"name\": \"final_acc\", \"run_id\": \"tf-mnist_1573636755_646589f7\", \"categories\": [0], \"series\": [{\"data\": [0.9725000262260437]}]}], \"run_logs\": \"2019-11-13 09:26:27,029|azureml|DEBUG|Inputs:: kwargs: {'OutputCollection': True, 'snapshotProject': True, 'only_in_process_features': True, 'skip_track_logs_dir': True}, track_folders: None, deny_list: None, directories_to_watch: []\\n2019-11-13 09:26:27,030|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Execution target type: batchai\\n2019-11-13 09:26:27,031|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Failed to import pyspark with error: No module named 'pyspark'\\n2019-11-13 09:26:27,031|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Pinning working directory for filesystems: ['pyfs']\\n2019-11-13 09:26:27,032|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,032|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Constructing mms service url in from history url environment variable None, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,033|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,096|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:27,102|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:27,113|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:27,119|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:27,125|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:27,131|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:27,132|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.RunClient.get-async:False|DEBUG|[START]\\n2019-11-13 09:26:27,132|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:27,132|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7'\\n2019-11-13 09:26:27,132|msrest.http_logger|DEBUG|Request method: 'GET'\\n2019-11-13 09:26:27,132|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '6a94ba6b-76c3-4965-9119-bc992018bd74'\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|    'request-id': '6a94ba6b-76c3-4965-9119-bc992018bd74'\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74'\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:27,133|msrest.http_logger|DEBUG|None\\n2019-11-13 09:26:27,133|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:27,133|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:27,133|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:27,133|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:27,185|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:27,185|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:27,185|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:27 GMT'\\n2019-11-13 09:26:27,185|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2019-11-13 09:26:27,185|msrest.http_logger|DEBUG|    'Transfer-Encoding': 'chunked'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'Vary': 'Accept-Encoding'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '6a94ba6b-76c3-4965-9119-bc992018bd74'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|    'Content-Encoding': 'gzip'\\n2019-11-13 09:26:27,186|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:27,187|msrest.http_logger|DEBUG|{\\n  \\\"runNumber\\\": 3,\\n  \\\"rootRunId\\\": \\\"tf-mnist_1573636755_646589f7\\\",\\n  \\\"experimentId\\\": \\\"6d2cea9b-101e-48d1-bc56-c7df925a2832\\\",\\n  \\\"createdUtc\\\": \\\"2019-11-13T09:19:17.6641762+00:00\\\",\\n  \\\"createdBy\\\": {\\n    \\\"userObjectId\\\": \\\"ff4137a8-2d37-4ee7-8675-9876eba469ce\\\",\\n    \\\"userPuId\\\": \\\"1003200026778A56\\\",\\n    \\\"userTenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\",\\n    \\\"userName\\\": \\\"Serge Retkowsky\\\"\\n  },\\n  \\\"userId\\\": \\\"ff4137a8-2d37-4ee7-8675-9876eba469ce\\\",\\n  \\\"token\\\": null,\\n  \\\"tokenExpiryTimeUtc\\\": null,\\n  \\\"error\\\": null,\\n  \\\"warnings\\\": null,\\n  \\\"revision\\\": 9,\\n  \\\"runId\\\": \\\"tf-mnist_1573636755_646589f7\\\",\\n  \\\"parentRunId\\\": null,\\n  \\\"status\\\": \\\"Running\\\",\\n  \\\"startTimeUtc\\\": \\\"2019-11-13T09:24:06.9249983+00:00\\\",\\n  \\\"endTimeUtc\\\": null,\\n  \\\"heartbeatEnabled\\\": false,\\n  \\\"options\\\": {\\n    \\\"generateDataContainerIdIfNotSpecified\\\": true\\n  },\\n  \\\"name\\\": null,\\n  \\\"dataContainerId\\\": \\\"dcid.tf-mnist_1573636755_646589f7\\\",\\n  \\\"description\\\": null,\\n  \\\"hidden\\\": false,\\n  \\\"runType\\\": \\\"azureml.scriptrun\\\",\\n  \\\"properties\\\": {\\n    \\\"_azureml.ComputeTargetType\\\": \\\"batchai\\\",\\n    \\\"ContentSnapshotId\\\": \\\"2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850\\\",\\n    \\\"AzureML.DerivedImageName\\\": \\\"azureml/azureml_75651667c3311d0b90ca821953173db3\\\",\\n    \\\"ProcessInfoFile\\\": \\\"azureml-logs/process_info.json\\\",\\n    \\\"ProcessStatusFile\\\": \\\"azureml-logs/process_status.json\\\"\\n  },\\n  \\\"scriptName\\\": \\\"tf_mnist.py\\\",\\n  \\\"target\\\": \\\"gpu-clustertf\\\",\\n  \\\"tags\\\": {\\n    \\\"_aml_system_ComputeTargetStatus\\\": \\\"{\\\\\\\"AllocationState\\\\\\\":\\\\\\\"steady\\\\\\\",\\\\\\\"PreparingNodeCount\\\\\\\":1,\\\\\\\"RunningNodeCount\\\\\\\":0,\\\\\\\"CurrentNodeCount\\\\\\\":1}\\\"\\n  },\\n  \\\"inputDatasets\\\": [\\n    {\\n      \\\"identifier\\\": {\\n        \\\"savedId\\\": \\\"3afaa0f2-1b8d-4b73-bf27-03d95e1d24f4\\\",\\n        \\\"registeredId\\\": null,\\n        \\\"registeredVersion\\\": null\\n      },\\n      \\\"consumptionType\\\": \\\"RunInput\\\",\\n      \\\"inputDetails\\\": {\\n        \\\"inputName\\\": \\\"mnist\\\",\\n        \\\"mechanism\\\": \\\"Mount\\\",\\n        \\\"pathOnCompute\\\": null\\n      }\\n    }\\n  ],\\n  \\\"runDefinition\\\": null,\\n  \\\"createdFrom\\\": null,\\n  \\\"cancelUri\\\": \\\"https://westeurope.experiments.azureml.net/execution/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runId/tf-mnist_1573636755_646589f7/cancel\\\",\\n  \\\"completeUri\\\": null,\\n  \\\"diagnosticsUri\\\": \\\"https://westeurope.experiments.azureml.net/execution/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runId/tf-mnist_1573636755_646589f7/diagnostics\\\"\\n}\\n2019-11-13 09:26:27,188|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.RunClient.get-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:27,189|azureml._SubmittedRun#tf-mnist_1573636755_646589f7|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'batchai', 'ContentSnapshotId': '2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850', 'AzureML.DerivedImageName': 'azureml/azureml_75651667c3311d0b90ca821953173db3', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2019-11-13 09:26:27,189|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2019-11-13 09:26:27,189|azureml|WARNING|Could not import azureml.mlflow or azureml.contrib.mlflow mlflow APIs will not run against AzureML services.  Add azureml-mlflow as a conda dependency for the run if this behavior is desired\\n2019-11-13 09:26:27,190|azureml.WorkerPool|DEBUG|[START]\\n2019-11-13 09:26:27,190|azureml.SendRunKillSignal|DEBUG|[START]\\n2019-11-13 09:26:27,190|azureml.RunStatusContext|DEBUG|[START]\\n2019-11-13 09:26:27,190|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunContextManager.RunStatusContext|DEBUG|[START]\\n2019-11-13 09:26:27,190|azureml.WorkingDirectoryCM|DEBUG|[START]\\n2019-11-13 09:26:27,190|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[START]\\n2019-11-13 09:26:27,190|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/azuremlservice/azureml/tf-mnist_1573636755_646589f7/mounts/workspaceblobstore/azureml/tf-mnist_1573636755_646589f7\\n2019-11-13 09:26:27,190|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2019-11-13 09:26:27,190|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Storing working dir for pyfs as /mnt/batch/tasks/shared/LS_root/jobs/azuremlservice/azureml/tf-mnist_1573636755_646589f7/mounts/workspaceblobstore/azureml/tf-mnist_1573636755_646589f7\\n2019-11-13 09:26:33,475|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,475|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,475|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,475|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,475|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,476|azureml._base_sdk_common.service_discovery|DEBUG|Constructing mms service url in from history url environment variable None, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,476|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,476|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,476|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westeurope.experiments.azureml.net.\\n2019-11-13 09:26:33,483|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:33,483|azureml._run_impl.run_history_facade|DEBUG|Created a static thread pool for RunHistoryFacade class\\n2019-11-13 09:26:33,490|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:33,496|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:33,502|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:33,508|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:33,508|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.RunClient.get-async:False|DEBUG|[START]\\n2019-11-13 09:26:33,509|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:33,509|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7'\\n2019-11-13 09:26:33,509|msrest.http_logger|DEBUG|Request method: 'GET'\\n2019-11-13 09:26:33,509|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:33,509|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:33,509|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2019-11-13 09:26:33,510|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '7175360a-e3ae-4056-9e48-e1feca73238e'\\n2019-11-13 09:26:33,510|msrest.http_logger|DEBUG|    'request-id': '7175360a-e3ae-4056-9e48-e1feca73238e'\\n2019-11-13 09:26:33,510|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74'\\n2019-11-13 09:26:33,510|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:33,510|msrest.http_logger|DEBUG|None\\n2019-11-13 09:26:33,510|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:33,510|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:33,510|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:33,510|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:33,566|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:33 GMT'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'Transfer-Encoding': 'chunked'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'Vary': 'Accept-Encoding'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '7175360a-e3ae-4056-9e48-e1feca73238e'\\n2019-11-13 09:26:33,567|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:33,568|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:33,568|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:33,568|msrest.http_logger|DEBUG|    'Content-Encoding': 'gzip'\\n2019-11-13 09:26:33,568|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:33,568|msrest.http_logger|DEBUG|{\\n  \\\"runNumber\\\": 3,\\n  \\\"rootRunId\\\": \\\"tf-mnist_1573636755_646589f7\\\",\\n  \\\"experimentId\\\": \\\"6d2cea9b-101e-48d1-bc56-c7df925a2832\\\",\\n  \\\"createdUtc\\\": \\\"2019-11-13T09:19:17.6641762+00:00\\\",\\n  \\\"createdBy\\\": {\\n    \\\"userObjectId\\\": \\\"ff4137a8-2d37-4ee7-8675-9876eba469ce\\\",\\n    \\\"userPuId\\\": \\\"1003200026778A56\\\",\\n    \\\"userTenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\",\\n    \\\"userName\\\": \\\"Serge Retkowsky\\\"\\n  },\\n  \\\"userId\\\": \\\"ff4137a8-2d37-4ee7-8675-9876eba469ce\\\",\\n  \\\"token\\\": null,\\n  \\\"tokenExpiryTimeUtc\\\": null,\\n  \\\"error\\\": null,\\n  \\\"warnings\\\": null,\\n  \\\"revision\\\": 9,\\n  \\\"runId\\\": \\\"tf-mnist_1573636755_646589f7\\\",\\n  \\\"parentRunId\\\": null,\\n  \\\"status\\\": \\\"Running\\\",\\n  \\\"startTimeUtc\\\": \\\"2019-11-13T09:24:06.9249983+00:00\\\",\\n  \\\"endTimeUtc\\\": null,\\n  \\\"heartbeatEnabled\\\": false,\\n  \\\"options\\\": {\\n    \\\"generateDataContainerIdIfNotSpecified\\\": true\\n  },\\n  \\\"name\\\": null,\\n  \\\"dataContainerId\\\": \\\"dcid.tf-mnist_1573636755_646589f7\\\",\\n  \\\"description\\\": null,\\n  \\\"hidden\\\": false,\\n  \\\"runType\\\": \\\"azureml.scriptrun\\\",\\n  \\\"properties\\\": {\\n    \\\"_azureml.ComputeTargetType\\\": \\\"batchai\\\",\\n    \\\"ContentSnapshotId\\\": \\\"2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850\\\",\\n    \\\"AzureML.DerivedImageName\\\": \\\"azureml/azureml_75651667c3311d0b90ca821953173db3\\\",\\n    \\\"ProcessInfoFile\\\": \\\"azureml-logs/process_info.json\\\",\\n    \\\"ProcessStatusFile\\\": \\\"azureml-logs/process_status.json\\\"\\n  },\\n  \\\"scriptName\\\": \\\"tf_mnist.py\\\",\\n  \\\"target\\\": \\\"gpu-clustertf\\\",\\n  \\\"tags\\\": {\\n    \\\"_aml_system_ComputeTargetStatus\\\": \\\"{\\\\\\\"AllocationState\\\\\\\":\\\\\\\"steady\\\\\\\",\\\\\\\"PreparingNodeCount\\\\\\\":1,\\\\\\\"RunningNodeCount\\\\\\\":0,\\\\\\\"CurrentNodeCount\\\\\\\":1}\\\"\\n  },\\n  \\\"inputDatasets\\\": [\\n    {\\n      \\\"identifier\\\": {\\n        \\\"savedId\\\": \\\"3afaa0f2-1b8d-4b73-bf27-03d95e1d24f4\\\",\\n        \\\"registeredId\\\": null,\\n        \\\"registeredVersion\\\": null\\n      },\\n      \\\"consumptionType\\\": \\\"RunInput\\\",\\n      \\\"inputDetails\\\": {\\n        \\\"inputName\\\": \\\"mnist\\\",\\n        \\\"mechanism\\\": \\\"Mount\\\",\\n        \\\"pathOnCompute\\\": null\\n      }\\n    }\\n  ],\\n  \\\"runDefinition\\\": null,\\n  \\\"createdFrom\\\": null,\\n  \\\"cancelUri\\\": \\\"https://westeurope.experiments.azureml.net/execution/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runId/tf-mnist_1573636755_646589f7/cancel\\\",\\n  \\\"completeUri\\\": null,\\n  \\\"diagnosticsUri\\\": \\\"https://westeurope.experiments.azureml.net/execution/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runId/tf-mnist_1573636755_646589f7/diagnostics\\\"\\n}\\n2019-11-13 09:26:33,569|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.RunClient.get-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:33,570|azureml._SubmittedRun#tf-mnist_1573636755_646589f7|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'batchai', 'ContentSnapshotId': '2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850', 'AzureML.DerivedImageName': 'azureml/azureml_75651667c3311d0b90ca821953173db3', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2019-11-13 09:26:33,570|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2019-11-13 09:26:36,354|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2019-11-13 09:26:36,354|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2019-11-13 09:26:36,354|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2019-11-13 09:26:37,355|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:37,355|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:37,355|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:37,356|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:37,356|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:37,356|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:37,357|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:37,357|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:37,357|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:37,358|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:37,359|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:37,359|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 0__log_batch to queue of approximate size: 0\\n2019-11-13 09:26:37,359|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:37,359|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:37,360|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:37,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:37,360|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:37,360|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:37,361|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:37,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:37,361|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:37,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:37,361|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:37,361|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:37,361|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '8cb82b3b-927b-48ae-a522-4872fadf306d'\\n2019-11-13 09:26:37,362|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:37,362|msrest.http_logger|DEBUG|    'request-id': '8cb82b3b-927b-48ae-a522-4872fadf306d'\\n2019-11-13 09:26:37,362|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:37,362|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:37,363|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:37,363|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"1139e783-75fa-4226-ab04-41528265b692\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:36.353982Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.8399999737739563}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"e5f1fac5-c5bc-47a2-b9d5-c55567956c39\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:36.354892Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9050999879837036}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:37,363|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:37,363|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:37,363|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:37,363|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:37,475|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:37,475|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:37,475|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:37 GMT'\\n2019-11-13 09:26:37,475|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '8cb82b3b-927b-48ae-a522-4872fadf306d'\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:37,476|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:37,477|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:38,355|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:38,355|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:38,356|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:38,356|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:38,356|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:38,356|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:38,356|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:38,356|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.1__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:38,357|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:38,358|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:38,358|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 1__log_batch to queue of approximate size: 1\\n2019-11-13 09:26:38,358|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:38,359|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:38,359|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:38,359|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:38,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:38,360|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:38,360|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:38,360|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:38,360|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:38,360|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:38,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:38,361|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:38,361|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:38,361|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '046f43b4-d47b-4dbb-97fe-1104ba56382c'\\n2019-11-13 09:26:38,361|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:38,361|msrest.http_logger|DEBUG|    'request-id': '046f43b4-d47b-4dbb-97fe-1104ba56382c'\\n2019-11-13 09:26:38,362|msrest.http_logger|DEBUG|    'Content-Length': '743'\\n2019-11-13 09:26:38,362|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:38,362|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:38,362|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"124edadf-655a-42e1-aac0-eec2a28e1545\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:38.15258Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"f4dcd1e8-6bd2-4a82-a0d9-ad644dcd7519\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:38.152715Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9207000136375427}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:38,362|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:38,362|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:38,362|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:38,362|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:38,467|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:38,467|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:38,467|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:38 GMT'\\n2019-11-13 09:26:38,467|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '046f43b4-d47b-4dbb-97fe-1104ba56382c'\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:38,468|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:38,469|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:40,355|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:40,356|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:40,356|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:40,356|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:40,356|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:40,356|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:40,357|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:40,357|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.2__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:40,357|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:40,359|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:40,359|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 2__log_batch to queue of approximate size: 2\\n2019-11-13 09:26:40,359|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:40,360|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:40,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:40,360|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:40,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:40,360|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:40,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:40,361|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:40,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:40,361|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:40,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:40,361|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:40,361|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:40,362|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'c4957552-8fc8-4423-802e-aba538766484'\\n2019-11-13 09:26:40,362|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:40,362|msrest.http_logger|DEBUG|    'request-id': 'c4957552-8fc8-4423-802e-aba538766484'\\n2019-11-13 09:26:40,362|msrest.http_logger|DEBUG|    'Content-Length': '742'\\n2019-11-13 09:26:40,362|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:40,363|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:40,363|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"9b2997d0-f8ba-4e86-b501-670211c36b9e\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:39.958701Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9399999976158142}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"639c9b50-62eb-40d8-9bfe-61731c05c7a3\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:39.95884Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.930899977684021}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:40,363|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:40,363|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:40,363|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:40,363|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:40,444|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:40,444|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:40 GMT'\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'c4957552-8fc8-4423-802e-aba538766484'\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:40,445|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:40,446|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:40,446|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:40,446|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:40,447|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:42,356|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:42,356|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:42,356|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:42,356|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:42,356|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:42,357|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:42,357|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:42,357|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:42,358|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.3__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:42,359|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:42,359|azureml.core.authentication|DEBUG|Time to expire 1813954.64045 seconds\\n2019-11-13 09:26:42,359|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:42,360|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 3__log_batch to queue of approximate size: 3\\n2019-11-13 09:26:42,360|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:42,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:42,360|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:42,361|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:42,361|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:42,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:42,361|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:42,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:42,362|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:42,362|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:42,362|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '0e2331b2-5ec4-4b7f-86d3-9446b57d1f68'\\n2019-11-13 09:26:42,362|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:42,362|msrest.http_logger|DEBUG|    'request-id': '0e2331b2-5ec4-4b7f-86d3-9446b57d1f68'\\n2019-11-13 09:26:42,362|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:42,362|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:42,363|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:42,363|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:42,363|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"6b24fba8-4ffb-42b2-8bfe-f97b3f15550d\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:41.735544Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9399999976158142}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"a5ee42d1-c02a-4b8c-b650-c3dd2d33b9e0\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:41.735828Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9397000074386597}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:42,363|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:42,363|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:42,363|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:42,363|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:42,448|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:42,448|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:42 GMT'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '0e2331b2-5ec4-4b7f-86d3-9446b57d1f68'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:42,449|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:42,450|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:44,356|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:44,356|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:44,357|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:44,357|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:44,357|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:44,357|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:44,358|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:44,358|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.4__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:44,358|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:44,359|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:44,359|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 4__log_batch to queue of approximate size: 4\\n2019-11-13 09:26:44,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:44,360|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:44,360|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:44,361|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:44,361|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:44,361|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:44,361|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:44,361|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:44,362|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:44,362|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:44,362|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:44,362|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:44,362|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:44,362|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '6198e3b0-cf82-4f83-9af4-d76fdef5f491'\\n2019-11-13 09:26:44,362|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:44,362|msrest.http_logger|DEBUG|    'request-id': '6198e3b0-cf82-4f83-9af4-d76fdef5f491'\\n2019-11-13 09:26:44,363|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:44,363|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:44,363|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:44,363|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"1876d9a7-4be0-42e0-b557-280762055a03\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:43.536483Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"f8e20582-468e-483d-b581-cb5e3722c7df\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:43.536617Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9455999732017517}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:44,363|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:44,363|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:44,364|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:44,364|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:44,464|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:44,464|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:44,464|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:44 GMT'\\n2019-11-13 09:26:44,464|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:44,464|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:44,464|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:44,465|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '6198e3b0-cf82-4f83-9af4-d76fdef5f491'\\n2019-11-13 09:26:44,465|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:44,465|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:44,465|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:44,465|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:44,465|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:44,466|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:45,361|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:45,361|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:45,362|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:45,362|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:45,362|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:45,362|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:45,362|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:45,363|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:45,363|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.5__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:45,364|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:45,364|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:45,364|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 5__log_batch to queue of approximate size: 5\\n2019-11-13 09:26:45,365|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:45,365|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:45,365|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:45,365|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:45,365|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:45,365|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:45,365|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:45,365|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:45,366|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:45,366|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:45,366|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '33629e65-ddc0-40b5-90d4-ed3f7e499cde'\\n2019-11-13 09:26:45,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:45,366|msrest.http_logger|DEBUG|    'request-id': '33629e65-ddc0-40b5-90d4-ed3f7e499cde'\\n2019-11-13 09:26:45,366|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:45,366|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:45,367|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:45,367|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:45,367|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"d9fc908e-e95e-411b-a102-7bda3f04ebbe\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:45.334868Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"893cc359-2283-4ab2-aad5-dcba1c707bae\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:45.335001Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9480000138282776}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:45,367|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:45,367|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:45,367|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:45,367|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:45,458|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:45,458|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:45,458|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:45 GMT'\\n2019-11-13 09:26:45,458|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:45,458|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:45,458|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:45,459|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '33629e65-ddc0-40b5-90d4-ed3f7e499cde'\\n2019-11-13 09:26:45,459|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:45,459|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:45,459|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:45,459|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:45,459|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:45,460|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:47,362|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:47,362|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:47,362|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:47,362|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:47,363|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:47,363|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:47,363|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:47,363|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.6__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:47,363|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:47,365|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:47,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 6__log_batch to queue of approximate size: 6\\n2019-11-13 09:26:47,365|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:47,365|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:47,366|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:47,366|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:47,366|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:47,367|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:47,367|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:47,367|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:47,367|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:47,367|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:47,367|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:47,368|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:47,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:47,368|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'e742cd93-7785-4671-b8b5-8d6c5e933729'\\n2019-11-13 09:26:47,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:47,368|msrest.http_logger|DEBUG|    'request-id': 'e742cd93-7785-4671-b8b5-8d6c5e933729'\\n2019-11-13 09:26:47,368|msrest.http_logger|DEBUG|    'Content-Length': '727'\\n2019-11-13 09:26:47,369|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:47,369|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:47,369|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"db393f1c-14cd-4e7d-88e3-9871c3dd24c9\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:47.123736Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 1.0}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"1fad266d-0041-4bc2-88db-a75839cdb3d5\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:47.12391Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.951200008392334}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:47,369|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:47,369|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:47,369|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:47,369|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:47,452|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:47 GMT'\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'e742cd93-7785-4671-b8b5-8d6c5e933729'\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:47,453|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:47,454|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:47,454|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:47,455|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:49,363|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:49,363|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:49,363|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:49,363|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:49,363|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:49,363|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:49,364|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:49,364|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:49,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.7__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:49,366|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:49,366|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:49,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 7__log_batch to queue of approximate size: 7\\n2019-11-13 09:26:49,367|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:49,367|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:49,367|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:49,367|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:49,367|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:49,367|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:49,368|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:49,368|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:49,368|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '06c1d946-1947-4b51-bb6c-b4a23f1587ce'\\n2019-11-13 09:26:49,368|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:49,368|msrest.http_logger|DEBUG|    'request-id': '06c1d946-1947-4b51-bb6c-b4a23f1587ce'\\n2019-11-13 09:26:49,368|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:49,369|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:49,369|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:49,369|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:49,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:49,370|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:49,370|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"48ae4731-7535-4c8d-bd68-e948e0027bac\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:48.915245Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9800000190734863}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"3388ded4-bce3-4976-86a8-bb4a065b5b1e\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:48.915381Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9556999802589417}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:49,370|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:49,370|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:49,370|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:49,370|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:49,493|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:49,493|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:49,493|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:49 GMT'\\n2019-11-13 09:26:49,493|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:49,493|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:49,493|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:49,494|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '06c1d946-1947-4b51-bb6c-b4a23f1587ce'\\n2019-11-13 09:26:49,494|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:49,494|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:49,494|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:49,494|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:49,494|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:49,495|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:51,363|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:51,363|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:51,364|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:51,364|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:51,364|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:51,364|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:51,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:51,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:51,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.8__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:51,366|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:51,366|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:51,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 8__log_batch to queue of approximate size: 8\\n2019-11-13 09:26:51,367|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:51,367|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:51,367|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:51,368|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:51,367|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:51,368|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:51,368|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:51,368|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:51,368|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:51,369|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '5597e2d6-f48b-4d3f-af91-50de31c11eeb'\\n2019-11-13 09:26:51,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:51,369|msrest.http_logger|DEBUG|    'request-id': '5597e2d6-f48b-4d3f-af91-50de31c11eeb'\\n2019-11-13 09:26:51,369|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:51,369|msrest.http_logger|DEBUG|    'Content-Length': '729'\\n2019-11-13 09:26:51,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:51,369|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:51,370|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:51,370|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"5e2ffeaf-b0db-4eda-977e-152bc108f50b\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:50.713249Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 1.0}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"9bcc7b72-585d-41ad-8e31-5348e58633e1\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:50.713382Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9580000042915344}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:51,370|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:51,370|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:51,370|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:51,370|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:51,458|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:51,458|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:51,458|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:51 GMT'\\n2019-11-13 09:26:51,458|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:51,458|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:51,458|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:51,459|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '5597e2d6-f48b-4d3f-af91-50de31c11eeb'\\n2019-11-13 09:26:51,459|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:51,459|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:51,459|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:51,459|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:51,459|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:51,460|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:53,364|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:53,364|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:53,364|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:53,364|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:53,364|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:53,364|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:53,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:53,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.9__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:53,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:53,367|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:53,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 9__log_batch to queue of approximate size: 9\\n2019-11-13 09:26:53,367|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:53,368|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:53,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:53,368|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:53,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:53,368|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:53,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:53,369|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:53,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:53,369|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:53,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:53,369|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:53,369|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:53,370|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'c36d1c4c-ed08-484e-bc65-2a5fcbeb8752'\\n2019-11-13 09:26:53,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:53,370|msrest.http_logger|DEBUG|    'request-id': 'c36d1c4c-ed08-484e-bc65-2a5fcbeb8752'\\n2019-11-13 09:26:53,370|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:53,370|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:53,371|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:53,371|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"4d01055c-ae74-443f-b4a7-6b82a6c5b3ba\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:52.506887Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9800000190734863}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"f01413b3-9449-419f-a70f-934e62da09ee\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:52.507021Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9602000117301941}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:53,371|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:53,371|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:53,371|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:53,371|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:53,456|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:53,456|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:53,456|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:53 GMT'\\n2019-11-13 09:26:53,456|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:53,456|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'c36d1c4c-ed08-484e-bc65-2a5fcbeb8752'\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:53,457|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:53,458|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:54,364|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:54,364|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:54,364|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:54,365|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:54,365|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:54,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:54,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:54,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:54,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.10__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:54,367|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:54,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:54,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 10__log_batch to queue of approximate size: 10\\n2019-11-13 09:26:54,368|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:54,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:54,368|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:54,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:54,369|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:54,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:54,369|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:54,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:54,369|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:54,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:54,369|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:54,369|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:54,370|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '5f369077-5e22-458c-a9da-5008108488ae'\\n2019-11-13 09:26:54,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:54,370|msrest.http_logger|DEBUG|    'request-id': '5f369077-5e22-458c-a9da-5008108488ae'\\n2019-11-13 09:26:54,370|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:26:54,370|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:54,370|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:54,370|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"0dd3d281-2650-4e9e-85b8-ff69b44e8207\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:54.285514Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9800000190734863}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"fe6ebf00-004c-4a5a-8312-e98e56777eae\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:54.285648Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9605000019073486}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:54,370|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:54,370|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:54,371|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:54,371|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:54,495|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:54,495|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:54,495|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:54 GMT'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '5f369077-5e22-458c-a9da-5008108488ae'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:54,496|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:54,497|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:56,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:56,365|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:56,365|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:56,365|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:56,365|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:56,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:56,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:56,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:56,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.11__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:56,366|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:56,368|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:56,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 11__log_batch to queue of approximate size: 11\\n2019-11-13 09:26:56,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:56,368|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:56,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:56,369|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:56,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:56,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:56,369|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:56,370|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:56,370|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:56,370|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:56,370|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:56,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:56,370|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '0695f89b-b9a0-4d3c-bd99-bb537213d8e0'\\n2019-11-13 09:26:56,371|msrest.http_logger|DEBUG|    'request-id': '0695f89b-b9a0-4d3c-bd99-bb537213d8e0'\\n2019-11-13 09:26:56,371|msrest.http_logger|DEBUG|    'Content-Length': '743'\\n2019-11-13 09:26:56,371|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:56,371|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:56,371|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"b0eb3db1-eb97-438f-963f-ff5c04bab572\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:56.07258Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9800000190734863}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"5d9d146f-328e-48f2-890a-0fe9fcf81ebd\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:56.072713Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9643999934196472}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:56,371|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:56,371|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:56,371|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:56,371|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:56,460|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:56,460|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:56,460|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:56 GMT'\\n2019-11-13 09:26:56,460|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:56,460|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '0695f89b-b9a0-4d3c-bd99-bb537213d8e0'\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:56,461|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:56,462|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:26:58,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:26:58,365|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:26:58,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:26:58,366|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:58,366|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:26:58,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:26:58,367|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:26:58,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:26:58,367|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:26:58,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.12__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:26:58,368|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:26:58,368|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:26:58,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 12__log_batch to queue of approximate size: 12\\n2019-11-13 09:26:58,369|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:26:58,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:26:58,369|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:26:58,370|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:26:58,370|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:26:58,370|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:26:58,370|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:26:58,370|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:26:58,370|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:26:58,371|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:26:58,371|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:26:58,371|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:26:58,371|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'dea41765-bd66-4b52-a615-3aead2478148'\\n2019-11-13 09:26:58,371|msrest.http_logger|DEBUG|    'request-id': 'dea41765-bd66-4b52-a615-3aead2478148'\\n2019-11-13 09:26:58,371|msrest.http_logger|DEBUG|    'Content-Length': '743'\\n2019-11-13 09:26:58,372|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:26:58,372|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:26:58,372|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"577811ff-6f9a-42e7-8b48-8b0e8af61b7b\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:57.862836Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"6b868698-032e-4652-905b-66dc5dc55520\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:57.86297Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9650999903678894}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:26:58,372|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:26:58,372|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:26:58,372|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:26:58,372|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:26:58 GMT'\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:26:58,463|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'dea41765-bd66-4b52-a615-3aead2478148'\\n2019-11-13 09:26:58,464|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:26:58,464|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:26:58,464|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:26:58,464|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:26:58,464|msrest.http_logger|DEBUG|\\n2019-11-13 09:26:58,465|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:00,365|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:00,366|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:00,366|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:27:00,366|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:00,366|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:00,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:00,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:00,367|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:00,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.13__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:00,368|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:00,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:00,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 13__log_batch to queue of approximate size: 13\\n2019-11-13 09:27:00,369|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:00,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:00,370|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:00,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:00,370|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:00,370|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:00,370|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:00,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:00,371|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:00,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:00,371|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '236ed3fa-3bf2-4230-a64b-ea46d94be463'\\n2019-11-13 09:27:00,371|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:00,371|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:00,371|msrest.http_logger|DEBUG|    'request-id': '236ed3fa-3bf2-4230-a64b-ea46d94be463'\\n2019-11-13 09:27:00,372|msrest.http_logger|DEBUG|    'Content-Length': '729'\\n2019-11-13 09:27:00,372|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:00,372|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:00,372|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"11101648-0d8b-444a-b26e-946aeb9985d6\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:59.651103Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 1.0}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"ba8b245e-42cd-4729-89cd-f407ff0dff34\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:26:59.651236Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9675999879837036}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:00,372|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:00,372|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:00,372|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:00,373|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:00,538|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:00,538|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:00,538|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:00 GMT'\\n2019-11-13 09:27:00,538|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '236ed3fa-3bf2-4230-a64b-ea46d94be463'\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:00,539|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:00,540|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:02,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:02,366|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:02,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:27:02,367|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:02,367|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:02,367|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:02,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:02,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:02,369|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:02,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.14__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:02,369|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:02,370|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:27:02,370|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 14__log_batch to queue of approximate size: 14\\n2019-11-13 09:27:02,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:02,370|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:02,371|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:02,371|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:02,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:02,371|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:02,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:02,371|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:02,372|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:02,372|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:02,372|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:02,372|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '686f20e2-c0b0-41d3-9551-3724168005e3'\\n2019-11-13 09:27:02,372|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:02,372|msrest.http_logger|DEBUG|    'request-id': '686f20e2-c0b0-41d3-9551-3724168005e3'\\n2019-11-13 09:27:02,373|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:27:02,373|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:02,373|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:02,373|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"2c5a1ba3-2920-4d58-ada5-7e17dcb0664b\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:01.433448Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"8991591f-aa34-4b9b-9349-fdd58c87bc70\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:01.433581Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9682999849319458}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:02,373|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:02,373|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:02,373|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:02,374|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:02,459|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:02,459|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:02,459|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:02 GMT'\\n2019-11-13 09:27:02,459|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:02,459|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '686f20e2-c0b0-41d3-9551-3724168005e3'\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:02,460|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:02,461|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:03,366|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:03,367|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:03,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:27:03,367|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:03,367|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:03,367|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:03,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:03,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:03,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.15__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:03,369|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:03,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:03,370|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 15__log_batch to queue of approximate size: 15\\n2019-11-13 09:27:03,370|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:03,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:03,370|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:03,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:03,371|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:03,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:03,371|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:03,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:03,371|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:03,371|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:03,371|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '7b14c598-cd5d-4473-bd95-73230e592f83'\\n2019-11-13 09:27:03,371|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:03,371|msrest.http_logger|DEBUG|    'request-id': '7b14c598-cd5d-4473-bd95-73230e592f83'\\n2019-11-13 09:27:03,372|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:03,372|msrest.http_logger|DEBUG|    'Content-Length': '744'\\n2019-11-13 09:27:03,372|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:03,372|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:03,372|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"d7e7344d-f317-4c0e-8a96-295c805d8285\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:03.232358Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9399999976158142}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"6d776166-e588-4551-875e-a25e9980f32f\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:03.232493Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9674000144004822}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:03,372|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:03,372|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:03,372|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:03,373|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:03 GMT'\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:03,450|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '7b14c598-cd5d-4473-bd95-73230e592f83'\\n2019-11-13 09:27:03,451|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:03,451|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:03,451|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:03,451|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:03,451|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:03,452|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:05,367|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:05,367|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:05,367|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:27:05,368|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:05,368|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:05,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:05,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:05,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:05,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.16__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:05,370|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:05,370|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:05,371|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 16__log_batch to queue of approximate size: 16\\n2019-11-13 09:27:05,371|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:27:05,371|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:05,371|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:05,371|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:05,372|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:05,372|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:05,372|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:05,372|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:05,372|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:05,373|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:05,373|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:05,373|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:05,373|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'f5fe5219-ffe5-4020-8322-d6e2ec2def37'\\n2019-11-13 09:27:05,373|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:05,373|msrest.http_logger|DEBUG|    'request-id': 'f5fe5219-ffe5-4020-8322-d6e2ec2def37'\\n2019-11-13 09:27:05,374|msrest.http_logger|DEBUG|    'Content-Length': '728'\\n2019-11-13 09:27:05,374|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:05,374|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:05,374|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"3a040ec5-c496-413e-9c1c-2770f3589763\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:05.027605Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 1.0}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"5a686574-3780-4966-9a05-8bb2eef1ca07\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:05.02774Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9700000286102295}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:05,374|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:05,374|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:05,374|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:05,374|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:05,486|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:05,486|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:05,486|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:05 GMT'\\n2019-11-13 09:27:05,486|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'f5fe5219-ffe5-4020-8322-d6e2ec2def37'\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:05,487|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:05,488|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:07,367|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:07,368|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:07,368|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:27:07,368|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:07,368|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:07,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:07,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:07,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.17__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:07,369|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:07,371|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:07,371|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 17__log_batch to queue of approximate size: 17\\n2019-11-13 09:27:07,371|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:07,371|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2019-11-13 09:27:07,372|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:07,372|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:07,372|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:07,372|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:07,373|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:07,373|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:07,373|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:07,373|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:07,373|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:07,373|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:07,373|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:07,373|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'cf4d511c-377d-418b-b827-6c35ba487fc5'\\n2019-11-13 09:27:07,374|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:07,374|msrest.http_logger|DEBUG|    'request-id': 'cf4d511c-377d-418b-b827-6c35ba487fc5'\\n2019-11-13 09:27:07,374|msrest.http_logger|DEBUG|    'Content-Length': '743'\\n2019-11-13 09:27:07,374|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:07,374|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:07,375|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"2da9d96c-6b13-4d33-976e-6dba1fb9e8ba\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:06.827993Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"318f296e-17c3-4192-83c3-438c49dcf021\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:06.828125Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.970300018787384}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:07,375|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:07,375|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:07,375|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:07,375|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:07,456|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:07 GMT'\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'cf4d511c-377d-418b-b827-6c35ba487fc5'\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:07,457|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:07,458|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:07,458|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:07,459|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:09,368|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:09,368|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:09,369|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 2.\\n2019-11-13 09:27:09,369|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:09,369|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:09,370|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:09,370|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:09,370|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:09,370|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.18__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:09,371|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:09,372|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:09,372|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 18__log_batch to queue of approximate size: 18\\n2019-11-13 09:27:09,372|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:09,372|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:09,373|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:09,373|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:09,373|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:09,373|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:09,373|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:09,374|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:09,374|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:09,374|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:09,374|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '6f51e5ec-553e-4f1a-b943-fd986d042238'\\n2019-11-13 09:27:09,374|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:09,374|msrest.http_logger|DEBUG|    'request-id': '6f51e5ec-553e-4f1a-b943-fd986d042238'\\n2019-11-13 09:27:09,374|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:09,374|msrest.http_logger|DEBUG|    'Content-Length': '742'\\n2019-11-13 09:27:09,375|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:09,375|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:09,375|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"dd58d916-3203-4607-870e-b2ad1e1cf45a\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:08.616618Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9399999976158142}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"76612509-0963-4adf-853e-8714b53e9291\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:08.616756Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.97079998254776}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:09,375|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:09,375|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:09,376|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:09,376|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:09,454|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:09,454|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:09,454|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:09 GMT'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '6f51e5ec-553e-4f1a-b943-fd986d042238'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:09,455|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:09,457|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:11,202|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2019-11-13 09:27:11,202|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/azuremlservice/azureml/tf-mnist_1573636755_646589f7/mounts/workspaceblobstore/azureml/tf-mnist_1573636755_646589f7\\n2019-11-13 09:27:11,202|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Reverting working dir from /mnt/batch/tasks/shared/LS_root/jobs/azuremlservice/azureml/tf-mnist_1573636755_646589f7/mounts/workspaceblobstore/azureml/tf-mnist_1573636755_646589f7 to /mnt/batch/tasks/shared/LS_root/jobs/azuremlservice/azureml/tf-mnist_1573636755_646589f7/mounts/workspaceblobstore/azureml/tf-mnist_1573636755_646589f7\\n2019-11-13 09:27:11,202|azureml.history._tracking.PythonWorkingDirectory|INFO|Working dir is already updated /mnt/batch/tasks/shared/LS_root/jobs/azuremlservice/azureml/tf-mnist_1573636755_646589f7/mounts/workspaceblobstore/azureml/tf-mnist_1573636755_646589f7\\n2019-11-13 09:27:11,203|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[STOP]\\n2019-11-13 09:27:11,203|azureml.WorkingDirectoryCM|DEBUG|[STOP]\\n2019-11-13 09:27:11,203|azureml._SubmittedRun#tf-mnist_1573636755_646589f7|INFO|complete is not setting status for submitted runs.\\n2019-11-13 09:27:11,203|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2019-11-13 09:27:11,203|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2019-11-13 09:27:11,203|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2019-11-13 09:27:11,203|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2019-11-13 09:27:11,203|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2019-11-13 09:27:11,204|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2019-11-13 09:27:11,204|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2019-11-13 09:27:11,204|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2019-11-13 09:27:11,204|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2019-11-13 09:27:11,204|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2019-11-13 09:27:11,204|azureml.RunStatusContext|DEBUG|[STOP]\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [].\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2019-11-13 09:27:11,205|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2019-11-13 09:27:11,205|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2019-11-13 09:27:11,206|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2019-11-13 09:27:11,206|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 3.\\n2019-11-13 09:27:11,206|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:11,206|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2019-11-13 09:27:11,206|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2019-11-13 09:27:11,206|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2019-11-13 09:27:11,207|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2019-11-13 09:27:11,207|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.19__log_batch|DEBUG|Using basic handler - no exception handling\\n2019-11-13 09:27:11,208|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2019-11-13 09:27:11,208|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2019-11-13 09:27:11,209|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 19__log_batch to queue of approximate size: 19\\n2019-11-13 09:27:11,209|msrest.http_logger|DEBUG|Request URL: 'https://westeurope.experiments.azureml.net/history/v1.0/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7/batch/metrics'\\n2019-11-13 09:27:11,209|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2019-11-13 09:27:11,209|msrest.http_logger|DEBUG|Request method: 'POST'\\n2019-11-13 09:27:11,209|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2019-11-13 09:27:11,209|msrest.http_logger|DEBUG|Request headers:\\n2019-11-13 09:27:11,209|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,210|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2019-11-13 09:27:11,210|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2019-11-13 09:27:11,210|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2019-11-13 09:27:11,210|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,210|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '45006bbc-4699-4179-aee3-8d1f45524231'\\n2019-11-13 09:27:11,210|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|\\n2019-11-13 09:27:11,210|msrest.http_logger|DEBUG|    'request-id': '45006bbc-4699-4179-aee3-8d1f45524231'\\n2019-11-13 09:27:11,210|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2019-11-13 09:27:11,210|msrest.http_logger|DEBUG|    'Content-Length': '1094'\\n2019-11-13 09:27:11,211|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2019-11-13 09:27:11,211|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1057-azure-x86_64-with-debian-stretch-sid) msrest/0.6.10 azureml._restclient/core.1.0.74 sdk_run'\\n2019-11-13 09:27:11,211|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2019-11-13 09:27:11,211|msrest.http_logger|DEBUG|Request body:\\n2019-11-13 09:27:11,211|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [AsyncTask(0__log_batch), AsyncTask(1__log_batch), AsyncTask(2__log_batch), AsyncTask(3__log_batch), AsyncTask(4__log_batch), AsyncTask(5__log_batch), AsyncTask(6__log_batch), AsyncTask(7__log_batch), AsyncTask(8__log_batch), AsyncTask(9__log_batch), AsyncTask(10__log_batch), AsyncTask(11__log_batch), AsyncTask(12__log_batch), AsyncTask(13__log_batch), AsyncTask(14__log_batch), AsyncTask(15__log_batch), AsyncTask(16__log_batch), AsyncTask(17__log_batch), AsyncTask(18__log_batch), AsyncTask(19__log_batch)].\\n2019-11-13 09:27:11,211|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"afe774c8-8c01-42a2-b8de-226a3f3df3bb\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:10.409072Z\\\", \\\"name\\\": \\\"training_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"training_acc\\\": 0.9599999785423279}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"training_acc\\\", \\\"name\\\": \\\"training_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"0d32da01-9001-44a3-b0cb-1593558dd3bf\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:10.409206Z\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"validation_acc\\\": 0.9725000262260437}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"validation_acc\\\", \\\"name\\\": \\\"validation_acc\\\", \\\"type\\\": \\\"float\\\"}]}}, {\\\"metricId\\\": \\\"2ddd5a23-bf00-45c4-a4e8-fce8bb8f5f73\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2019-11-13T09:27:10.446179Z\\\", \\\"name\\\": \\\"final_acc\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"final_acc\\\": 0.9725000262260437}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"final_acc\\\", \\\"name\\\": \\\"final_acc\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2019-11-13 09:27:11,211|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,212|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2019-11-13 09:27:11,212|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,212|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2019-11-13 09:27:11,212|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,212|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2019-11-13 09:27:11,212|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.1__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,212|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2019-11-13 09:27:11,212|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.1__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,215|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.1__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,215|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.2__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,215|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.2__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,215|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.2__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.3__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.3__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.3__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.4__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.4__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.4__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.5__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.5__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.5__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.6__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.6__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.6__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,216|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.7__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.7__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.7__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.8__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.8__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.8__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.9__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.9__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.9__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.10__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.10__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.10__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.11__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,217|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.11__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.11__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.12__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.12__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.12__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.13__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.13__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.13__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.14__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.14__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.14__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.15__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.15__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,218|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.15__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.16__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.16__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.16__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.17__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.17__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.17__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.18__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.18__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,219|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.18__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|Response status: 200\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|Response headers:\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'Date': 'Wed, 13 Nov 2019 09:27:11 GMT'\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:6a27ce65-5555-41a3-85f7-b7a1ce31fd6b'\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '45006bbc-4699-4179-aee3-8d1f45524231'\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2019-11-13 09:27:11,303|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2019-11-13 09:27:11,304|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2019-11-13 09:27:11,304|msrest.http_logger|DEBUG|Response content:\\n2019-11-13 09:27:11,304|msrest.http_logger|DEBUG|\\n2019-11-13 09:27:11,304|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2019-11-13 09:27:11,470|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.19__log_batch.WaitingTask|DEBUG|[START]\\n2019-11-13 09:27:11,470|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.19__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2019-11-13 09:27:11,470|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.19__log_batch.WaitingTask|DEBUG|[STOP]\\n2019-11-13 09:27:11,470|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Waiting on task: 19__log_batch.\\n1 tasks left. Current duration of flush 0.00849461555480957 seconds.\\n\\n2019-11-13 09:27:11,470|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2019-11-13 09:27:11,470|azureml._SubmittedRun#tf-mnist_1573636755_646589f7.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2019-11-13 09:27:11,470|azureml.SendRunKillSignal|DEBUG|[STOP]\\n2019-11-13 09:27:11,471|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[START]\\n2019-11-13 09:27:11,471|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[STOP]\\n2019-11-13 09:27:11,471|azureml.WorkerPool|DEBUG|[STOP]\\n2019-11-13 09:27:11,566|azureml.data.context_managers|DEBUG|Enter __exit__ of DatasetContextManager\\n2019-11-13 09:27:11,566|azureml.data.context_managers|DEBUG|Unmounting /Dataset/mnist.\\n2019-11-13 09:27:11,641|azureml.data.context_managers|DEBUG|Finishing unmounting /Dataset/mnist.\\n2019-11-13 09:27:11,641|azureml.data.context_managers|DEBUG|Exit __exit__ of DatasetContextManager\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": true, \"log_level\": \"INFO\", \"sdk_version\": \"1.0.72\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "\n",
    "RunDetails(run).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also periodically check the status of the run object, and navigate to Azure portal to monitor the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>tf-mnist</td><td>tf-mnist_1573636755_646589f7</td><td>azureml.scriptrun</td><td>Queued</td><td><a href=\"https://ml.azure.com/experiments/tf-mnist/runs/tf-mnist_1573636755_646589f7?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlserviceresourcegroup/workspaces/azuremlservice\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.script_run.ScriptRun?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: tf-mnist,\n",
       "Id: tf-mnist_1573636755_646589f7,\n",
       "Type: azureml.scriptrun,\n",
       "Status: Queued)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Run object <a class=\"anchor\" id=\"run-object\"></a>\n",
    "The Run object provides the interface to the run history -- both to the job and to the control plane (this notebook), and both while the job is running and after it has completed. It provides a number of interesting features for instance:\n",
    "* `run.get_details()`: Provides a rich set of properties of the run\n",
    "* `run.get_metrics()`: Provides a dictionary with all the metrics that were reported for the Run\n",
    "* `run.get_file_names()`: List all the files that were uploaded to the run history for this Run. This will include the `outputs` and `logs` folder, azureml-logs and other logs, as well as files that were explicitly uploaded to the run using `run.upload_file()`\n",
    "\n",
    "Below are some examples -- please run through them and inspect their output. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'runId': 'tf-mnist_1573636755_646589f7',\n",
       " 'target': 'gpu-clustertf',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2019-11-13T09:24:06.924998Z',\n",
       " 'endTimeUtc': '2019-11-13T09:27:34.11495Z',\n",
       " 'properties': {'_azureml.ComputeTargetType': 'batchai',\n",
       "  'ContentSnapshotId': '2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850',\n",
       "  'AzureML.DerivedImageName': 'azureml/azureml_75651667c3311d0b90ca821953173db3',\n",
       "  'ProcessInfoFile': 'azureml-logs/process_info.json',\n",
       "  'ProcessStatusFile': 'azureml-logs/process_status.json'},\n",
       " 'inputDatasets': [{'dataset': {'id': '3afaa0f2-1b8d-4b73-bf27-03d95e1d24f4'}, 'consumptionDetails': {'type': 'RunInput', 'inputName': 'mnist', 'mechanism': 'Mount'}}],\n",
       " 'runDefinition': {'script': 'tf_mnist.py',\n",
       "  'arguments': ['--data-folder',\n",
       "   'DatasetConsumptionConfig:mnist',\n",
       "   '--batch-size',\n",
       "   '50',\n",
       "   '--first-layer-neurons',\n",
       "   '300',\n",
       "   '--second-layer-neurons',\n",
       "   '100',\n",
       "   '--learning-rate',\n",
       "   '0.01'],\n",
       "  'sourceDirectoryDataStore': None,\n",
       "  'framework': 'Python',\n",
       "  'communicator': 'None',\n",
       "  'target': 'gpu-clustertf',\n",
       "  'dataReferences': {},\n",
       "  'data': {'mnist': {'dataLocation': {'dataset': {'id': '3afaa0f2-1b8d-4b73-bf27-03d95e1d24f4'},\n",
       "     'dataPath': None},\n",
       "    'createOutputDirectories': False,\n",
       "    'mechanism': 'Mount',\n",
       "    'environmentVariableName': 'mnist',\n",
       "    'pathOnCompute': None,\n",
       "    'overwrite': False}},\n",
       "  'jobName': None,\n",
       "  'maxRunDurationSeconds': None,\n",
       "  'nodeCount': 1,\n",
       "  'environment': {'name': 'Experiment tf-mnist Environment',\n",
       "   'version': 'Autosave_2019-11-13T09:19:16Z_88de4c67',\n",
       "   'python': {'interpreterPath': 'python',\n",
       "    'userManagedDependencies': False,\n",
       "    'condaDependencies': {'channels': ['conda-forge'],\n",
       "     'dependencies': ['python=3.6.2',\n",
       "      {'pip': ['azureml-dataprep[pandas,fuse]',\n",
       "        'azureml-defaults',\n",
       "        'tensorflow-gpu==1.13.1',\n",
       "        'horovod==0.16.1']}],\n",
       "     'name': 'azureml_87bf8b18f50c3819153ad5fad9efbcd9'},\n",
       "    'baseCondaEnvironment': None},\n",
       "   'environmentVariables': {'EXAMPLE_ENV_VAR': 'EXAMPLE_VALUE'},\n",
       "   'docker': {'baseImage': 'mcr.microsoft.com/azureml/base-gpu:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04',\n",
       "    'baseDockerfile': None,\n",
       "    'baseImageRegistry': {'address': None, 'username': None, 'password': None},\n",
       "    'enabled': True,\n",
       "    'arguments': []},\n",
       "   'spark': {'repositories': [], 'packages': [], 'precachePackages': False},\n",
       "   'inferencingStackVersion': None},\n",
       "  'history': {'outputCollection': True,\n",
       "   'directoriesToWatch': ['logs'],\n",
       "   'snapshotProject': True},\n",
       "  'spark': {'configuration': {'spark.app.name': 'Azure ML Experiment',\n",
       "    'spark.yarn.maxAppAttempts': '1'}},\n",
       "  'amlCompute': {'name': None,\n",
       "   'vmSize': None,\n",
       "   'retainCluster': False,\n",
       "   'clusterMaxNodeCount': 1},\n",
       "  'tensorflow': {'workerCount': 1, 'parameterServerCount': 1},\n",
       "  'mpi': {'processCountPerNode': 1},\n",
       "  'hdi': {'yarnDeployMode': 'Cluster'},\n",
       "  'containerInstance': {'region': None, 'cpuCores': 2, 'memoryGb': 3.5},\n",
       "  'exposedPorts': None,\n",
       "  'docker': {'useDocker': True,\n",
       "   'sharedVolumes': True,\n",
       "   'shmSize': '2g',\n",
       "   'arguments': []}},\n",
       " 'logFiles': {'azureml-logs/55_azureml-execution-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/55_azureml-execution-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt?sv=2019-02-02&sr=b&sig=SndNgwFyDW7pnzg2od64ryD6f1GE9C647a5e3gVyNf8%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'azureml-logs/65_job_prep-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/65_job_prep-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt?sv=2019-02-02&sr=b&sig=bKMejKCJQCMpKrbUWjswNBQDSCK4kO7eznt%2BO2drhH8%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'azureml-logs/70_driver_log.txt': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=y3GL2k2z%2BqajjYVmYoAvY0OOxTcEtoXq%2FyJIrxylAWI%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'azureml-logs/75_job_post-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/75_job_post-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt?sv=2019-02-02&sr=b&sig=ERlhPMKoqMOZGz2J7oEHSYC4DBkb5A9BQVMaJ%2Bt%2Byc0%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'azureml-logs/process_info.json': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=YR5F3dKtMnsOFqd9LU0QoNvAx7QETpLrrVaFKPTGs04%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'azureml-logs/process_status.json': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=L5r7jyVpp2XtTPseWM5nrEG0D7YaNOxp1G%2FdMX39vrk%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'logs/azureml/140_azureml.log': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/logs/azureml/140_azureml.log?sv=2019-02-02&sr=b&sig=s4mqyAnc9PSgTWFORV7v9hUJerGN89qgF56s%2F1Mq0hc%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r',\n",
       "  'logs/azureml/azureml.log': 'https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573636755_646589f7/logs/azureml/azureml.log?sv=2019-02-02&sr=b&sig=dwtSL%2B3xLILJph2NEyySdQks5U1c2Kc7o05PCH6NTX4%3D&st=2019-11-13T09%3A18%3A12Z&se=2019-11-13T17%3A28%3A12Z&sp=r'}}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run.get_details()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'training_acc': [0.8399999737739563,\n",
       "  0.9599999785423279,\n",
       "  0.9399999976158142,\n",
       "  0.9399999976158142,\n",
       "  0.9599999785423279,\n",
       "  0.9599999785423279,\n",
       "  1,\n",
       "  0.9800000190734863,\n",
       "  1,\n",
       "  0.9800000190734863,\n",
       "  0.9800000190734863,\n",
       "  0.9800000190734863,\n",
       "  0.9599999785423279,\n",
       "  1,\n",
       "  0.9599999785423279,\n",
       "  0.9399999976158142,\n",
       "  1,\n",
       "  0.9599999785423279,\n",
       "  0.9399999976158142,\n",
       "  0.9599999785423279],\n",
       " 'validation_acc': [0.9050999879837036,\n",
       "  0.9207000136375427,\n",
       "  0.930899977684021,\n",
       "  0.9397000074386597,\n",
       "  0.9455999732017517,\n",
       "  0.9480000138282776,\n",
       "  0.951200008392334,\n",
       "  0.9556999802589417,\n",
       "  0.9580000042915344,\n",
       "  0.9602000117301941,\n",
       "  0.9605000019073486,\n",
       "  0.9643999934196472,\n",
       "  0.9650999903678894,\n",
       "  0.9675999879837036,\n",
       "  0.9682999849319458,\n",
       "  0.9674000144004822,\n",
       "  0.9700000286102295,\n",
       "  0.970300018787384,\n",
       "  0.97079998254776,\n",
       "  0.9725000262260437],\n",
       " 'final_acc': 0.9725000262260437}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run.get_metrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['azureml-logs/55_azureml-execution-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt',\n",
       " 'azureml-logs/65_job_prep-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt',\n",
       " 'azureml-logs/70_driver_log.txt',\n",
       " 'azureml-logs/75_job_post-tvmps_1dd08aab37c0648f3aee163129f022f246c0287e9ee1be8ad1f0bd82afbdcc45_d.txt',\n",
       " 'azureml-logs/process_info.json',\n",
       " 'azureml-logs/process_status.json',\n",
       " 'logs/azureml/140_azureml.log',\n",
       " 'logs/azureml/azureml.log',\n",
       " 'outputs/model/checkpoint',\n",
       " 'outputs/model/mnist-tf.model.data-00000-of-00001',\n",
       " 'outputs/model/mnist-tf.model.index',\n",
       " 'outputs/model/mnist-tf.model.meta']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run.get_file_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot accuracy over epochs\n",
    "Since we can retrieve the metrics from the run, we can easily make plots using `matplotlib` in the notebook. Then we can add the plotted image to the run using `run.log_image()`, so all information about the run is kept together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxMAAAFTCAYAAABCnVEVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXyU1dn4/89JCIQlEJIgyL7JTggQNgFBQEDBFRcEF1TELmrbX7XV72Nrf/Z5Wtvafp9abbUCbiCCoBQVFxYBEUFAAyiLshMgrCEJkECW8/3jmiGTyWSfzD3L9X697tcs9z33nJlJZs51lusYay1KKaWUUkopVVVRThdAKaWUUkopFZo0mFBKKaWUUkpViwYTSimllFJKqWrRYEIppZRSSilVLRpMKKWUUkoppapFgwmllFJKKaVUtWgwoZRSVWCMmWmMscaYvzldFuUfxpjXXJ+pr22xw2Xbb4yZ42QZlFKqPHWcLoBSSoUKY0x94DbXzanGmF9ZawucLJPymxPADT7uPx3ogiilVCjRYEIppSrvZqAxsBS4DhgPfOBoiXwwxtSz1l5wuhzBpBLvyUVr7fqAFUgppcKEDnNSSqnKuxfIBKYBucA9vg4yxvQxxrxnjDlljMk1xuwyxjzpdczNxpgvjDFnjTHZxpivjDE3uPa1dw2xmeb1mJGu+0d63LfKGLPWGHO9MeYbY8wF4CeufQ8bY740xpw2xpwxxqw3xkzwUd6GxphnjTF7jDEXjDEZxphFxpjmxpj+rue80cfjXjPGpBtjost6w4z4hes9uGiMOWqMecEY09jjmO+MMYt8PHaQ67lv8npvlxhjMl3v7RfGmOFllGuIMWadMSYX+HNZZawsj/NeaYzZaIzJcw1DesTHsQONMctdn+85Y8wKY8xAH8eNMMYsM8ZkuY7bYox5wMdxk40xO1zHbDLGDPPaP8B1nlPGmPPGmL3GmH/W9DUrpVRFNJhQSqlKMMa0BMYA8621J4DFwA3GmKZexw0EvgQ6Ab8AJgB/A1p7HPMI8C5wHAlQbgPeA9pXs3hdgOeBfwDjgBWu+9sDM13nvwPYBHxgjLnWoyx1gWXAo8BrwETgYWR4T1Nr7WZgI/CQ1+uMB24HZlprC8sp2/8gr38ZcD1SqZ8GfGiMcf8GvQlM9H4vgbtc5Vjqes5+wDogAXgQmAScApYbY/p7PbYJ8DYwD7gWeKucMrpfUx0fm/E6rDEwH3gduAlYBTzvGfgZY5KB1UBT12u9x/W41caYPh7H3Yh8VnWR9/dGYDbQzus5hwO/BH6DfI7RyOcY7zpPI+AToND1fNcBz6CjD5RSgWCt1U033XTTrYIN+DVggSGu2+Nct3/kddwa4BDQoIzzNAZygHfLea72rnNP87p/pOv+kR73rQKKgJQKyh+FVC4/Bf7jcf/9rnPeUM5jpyEV1XYe9z0KFACty3lcApAHvOZ1/12ezwm0cZ3/IY9jYpB5DP/0uG8FsAOo63FftOu+xR73veY6/42V/Gzdx/vaHvNx3GSvxy8DDgDGdXshcAaI9/rcT7s/d8AA+5EAL6qcsu1HesOaetyX6irHFK/byU7/n+imm26Rt2nPhFJKVc49wA/W2i9dt5cDR/AY6mSMaQAMBeZaa8+XcZ4rgUbAv/1Ytv3W2jTvO11DlD4wxhxDKv75wDVAV4/DxgIZ1tol5Zz/baRy/KDHfQ8BH1pr08t53GCgHuCdjehtV3lGAFhrDyEt+Xd7HDMeSALecL2W+q7j3wGK3D0HSKV8OXCV13MUULX5LMeBAT62N72OKwS8h2S9DbQFWrluXwV8YK094z7AWpsNLHG/ZuQzaIf07BRVULYvrbWZHre3uS7bui5/QD6fl40xdxlj2lRwPqWU8hsNJpRSqgLGmAFAD+BdY0y8a3hJHDJUaYgxpovr0KbI92p5FexE12V5x1TVUe87XBXKFUjvwCNIEDMA+BiI9SrP4fJObq3NA14FHnBV4ocj78dLFZQrwVf5rGTAOuWxHyRoGGqM6eC6fTew2xZPik5AeiF+gwRFntvDQFOPYVMAx235w6+85VtrN/nYjnkdl2mtzfe6z32MO5hI8H7NLhnI3whU7e+gREYpWzyRPNZ1Owu4Gglu/wkcNMZ8a4yZVIlzK6VUjWgwoZRSFbvXdflrZMiJe3vYdb+7dyITGXLUirKddF2Wd0ye67Ku1/2J3ge6WB/3jUfmDdxurV1grV1vrd0ENPBRnvLK4vYvoDkyrv8hZPjNJxU8xl0JbuF5p6tHIREJKNwWAeeBu1yTs6+nZK/AGeS9/Qe+exAGeLXw+3pP/KGpMSbG677mrkt3UHYar9fs0oLi96QyfweVZq1Ns9ZOQgKZIcAeYIExppc/zq+UUmXRYEIppcrhmqA8GdiAtP56b2nA3cYY4xratBapENcv45TrgLPAjHKe9hhwAfCuCJbKxFQOd9BwqRXd1YMy1Ou4T4EWxpjryzuZtXaP69jHgVuBVyoxPGc98jome91/BzJ/Y7XH+XOA/yA9Erchre5veuw/B3wO9AG+9tWLUEFZ/CUamfjtaTJwkOJgYjUwwRgT5z7Adf16il/z90hANt3HJO9qs9YWuHpzfoP8xnf317mVUsoXzfSglFLlm4i0ov/SWrvKe6cx5mWk1X4k8BnwGFJh/NIY81dkGEtHZIL0I9baHCNpYv/hSoc6F5mQnQLkWWv/Ya21xpj5yLCi74FdSCAxsgrlXo7MG3jDVY7Lgf8fqfR6NiTNQeZCzDPG/BEJmuKQCeb/a63d6XHsP5EKfz6Sdahc1trTRlYKf9IYcw7JytQd+G8k6PrQ6yFvAHe6yrnWWrvPa///h0xw/8QYMwsZSpQE9AOirbVPVFSmctQ1xgz2cf95a+1Wj9s5wJ+NMUnIXIU7kSxf06y17t6Q3yN/NyuMMX9Cekl+jQR4zwC4PuOfI0PlVhpjXkImnHcHLrPWPl3ZghtjJiLB6WJgH9AQmSCfg2QWU0qp2uP0DHDddNNNt2DekMpzNmVnZ2qCDM95zeO+vsD7yNCcXGAn8Guvx92KVNxzXeffAEz02B+PtMyfRIbGvIQEFL6yOa0to2y3u547D/gOaUF/DZmw7XlcI+AvSEaii0glfSFSqfU8Lho4B7xThffPIClyd3mc+0WgsY9jo137LTCjjPN1RyY8H0d6PdKRic3XeRzzGpBehTK+RtnZnL71Pi8y/2Sj6309ADzq45yDkIDurOs9WwEM9HHcKCQIPevatgD3eezfD8zx8TgL/M51vSuSrnafq0wnkMBtkNP/P7rpplv4b+40dkoppVS5jDHXIEOdxlhrV1R0fLgxxryGvPbWFR2rlFKRQoc5KaWUKpcxphMyVOv/IvMVIi6QUEop5ZtOwFZKKVWR3wAfIcOK7qngWKWUUhFEhzkppZRSSimlqkV7JpRSSimllFLVosGEUkoppZRSqlrCdgJ2UlKSbd++vdPFUEoppZRSKqRt3rz5pLW2ma99YRtMtG/fnk2bArUgqlJKKaWUUuHJGHOgrH06zEkppZRSSilVLRpMKKWUUkoppapFgwmllFJKKaVUtYTtnAlf8vPzSU9PJy8vz+miqDAUGxtL69atiYmJcbooSimllFIBEVHBRHp6OnFxcbRv3x5jjNPFUWHEWsupU6dIT0+nQ4cOThdHKaWUUiogImqYU15eHomJiRpIKL8zxpCYmKi9XkoppZSKKBEVTAAaSKhao39bSimllIo0AQsmjDGzjTHHjTHflrHfGGOeN8bsNsZsNcb089h3rzHmB9d2b6DKXBuio6NJSUm5tO3fv7/c49u3b8/JkycBaNSoUa2WzfO5qmrx4sVs37790u3f/va3LF++3F9Fq7Q//OEPAX9OpZRSSqlIFcieideA8eXsvxa4wrXNAP4FYIxJAJ4GBgEDgaeNMU1rtaS1qH79+qSlpV3awmWVbu9g4plnnmHMmDEBL4cGE0oppZRSgROwYMJauwY4Xc4hNwJvWLEeiDfGXA6MA5ZZa09bazOBZZQflISc1157jYcffvjS7YkTJ7Jq1apKP37OnDkMHDiQlJQUHnroIQoLC/nXv/7Fr371qxLP8cgjjwBw00030b9/f3r27Mm///3vUufbv38/vXr1unT7ueee43e/+x0Ar7zyCgMGDKBPnz5MmjSJ8+fPs27dOpYsWcLjjz9OSkoKe/bsYdq0aSxcuBCAFStW0LdvX3r37s3999/PhQsXAOkJefrpp+nXrx+9e/dm586dpcry3XffXXptycnJ/PDDD2W+5ieeeILc3FxSUlKYOnVqpd8/FZz27YPcXKdLEVysdboESlWNtXDgABQUOF0SVZb9++H8eadLoUJZMGVzagUc8rid7rqvrPtLMcbMQHo1aNu2bdnP9NBDNStpRV5+ucxd7souQIcOHXjvvfdq9FQ7duxg/vz5fPHFF8TExPCTn/yEuXPncuuttzJkyBD+/Oc/AzB//nz+67/+C4DZs2eTkJBAbm4uAwYMYNKkSSQmJlbq+W655RYefPBBAJ566ilmzZrFI488wg033MDEiRO59dZbSxyfl5fHtGnTWLFiBV26dOGee+7hX//6Fz//+c8BSEpK4uuvv+af//wnzz33HDNnzizx+Jdeeomf/exnTJ06lYsXL1JYWFjma3722Wd54YUXSEtLq9F7qpyXnw8ffSQ/cNOnQ3y80yVy3t69sG4d3HEHaPZhFSrOnoVXX4WUFLjpJqdLo7xlZcHrr0OXLnDnnU6XRoWqYJqA7Wv2qi3n/tJ3Wvtva22qtTa1WbNmfi2cv3gOc6ppIAHS6r9582YGDBhASkoKK1asYO/evTRr1oyOHTuyfv16Tp06xa5duxg6dCgAzz//PH369GHw4MEcOnToUmt/ZXz77bcMHz6c3r17M3fuXL777rtyj9+1axcdOnSgS5cuANx7772sWbPm0v5bbrkFgP79+/ucPzJkyBD+8Ic/8Kc//YkDBw5Qv379Ml+zCh8xMTBhgrRmzpwJGRlOl8hZW7fCnDmQnQ0XLsj7sWaN9lSo4BcXJ4FEWhpUc0qeqkVbtsj3yK5d0husVHUEUzCRDrTxuN0aOFLO/WGjTp06FBUVXbpdlfSi1lruvffeSwHKrl27Lg1JuuOOO1iwYAGLFi3i5ptvxhjDqlWrWL58OV9++SVbtmyhb9++pZ6vvPJMmzaNF154gW3btvH0009XWFZbQW2nXr16gExML/DRDz5lyhSWLFlC/fr1GTduHCtXriz3NavQt2cPnD4N7drB/fdDdLS0bEZivGgtfPEFvPsutG0r70ejRrBtG6xcCUuWQGGh06VUyrcjR6RB4JproF49WLbM6RIpb127yucTHw+ffKINFKp6gimYWALc48rqNBjIstYeBT4BxhpjmromXo913Rc22rdvT1paGkVFRRw6dIivvvqq0o8dPXo0Cxcu5Pjx4wCcPn2aAwcOANLqv3jxYubNm8cdd9wBQFZWFk2bNqVBgwbs3LmT9evXlzpn8+bNOX78OKdOneLChQt88MEHl/bl5ORw+eWXk5+fz9y5cy/dHxcXR05OTqlzdevWjf3797N7924A3nzzTUaMGFHp17d37146duzIo48+yg033MDWrVvLfc0xMTHk5+dX+vwquFy4IBXn99+X25ddBg88ID90a9dG3g/d559LBaxXL7jrLoiNlfvHjIERI+Cbb+Dtt+HiRWfLqZS33Fx480344ANo2BCGD9fW72DUvDkMHQoTJ8LYsaAZzlV1BGzOhDFmHjASSDLGpCMZmmIArLUvAUuB64DdwHngPte+08aY3wMbXad6xlpb3kTuipUzp8EJQ4cOpUOHDvTu3ZtevXrRr1+/ih/k0qNHD/77v/+bsWPHUlRURExMDC+++CLt2rWjadOm9OjRg+3btzNw4EAAxo8fz0svvURycjJdu3Zl8ODBpc4ZExPDb3/7WwYNGkSHDh3o1q3bpX2///3vGTRoEO3ataN3796XAojJkyfz4IMP8vzzz1+aeA0QGxvLq6++ym233UZBQQEDBgzgRz/6UaVf3/z585kzZw4xMTG0aNGC3/72tyQkJJT5mmfMmEFycjL9+vUrEeyo0LB2LZw7J5Vlt8aN4b77JJAwRlo6o6Mj40evZ0/peRg5suTrNQauvlremw8+gNdeg7vvhvr1nSqpUiWtXg15eXDllXJ78GAZrpeV5Wy5VLG0NEhIkF7Pzp2dLo0KZaaiYSihKjU11W7atKnEfTt27KB79+4OlUhFAv0bq74zZ+CFF6BHD3BNpSmloEBaO1u0gHHjICqY+lb9JDdXehyGDKlcwLRrlxx/220SZCnltFOn4MUXoV8/afF2KyoKz//ZUJSfD889B927F0+MtxY+/VQaJa66ytnyqeBjjNlsrU31tU//rZVSQWHFCrkcPbrsY6KjoVUr2LAB3nlHfhDDSVYWzJ4t74VrFF+FunaFyZPlvTl3TsapK+WkZcskicLIkSXvj4qSCuv27To0z2k7d8qwUldySUAaL3JyZHhldrZzZVOhR4MJpZTjrIUGDWDYMGjSpOzjjJFxvePHy4/hm2+Gz1oUx45J5qqcHBmy1Lx51c+xdKlMVv/+e/+XT6nKuHhR0sEOGybJArwdOwYLFkiaY+WctDSZi9auXcn7x4yR72N3445SlaHBhFLKccbAtdeWbsksy+DBcOutcPgwLFpUq0ULiH37pEfCGMnY1L599c5z7bXQrJlMyv76a78WUalKqVtXkia450p4a9FC5gJ98YW2fjslK0uy46WklB5KGR8v369btmgvp6o8DSaUUo7au1dWyK2qnj2lBX/cOP+XKdAKC6FpU6mEXXZZ9c/TqBFMmwYdO0ra2NWrIy8DlnLOoUMy1M6Y8ufvjBkj8ydWrgxc2VSxkyelJ7hPH9/7hw2T/cuXB7ZcKnRpMKGUckxBgaSBXbq0epXe9u2lJd49cTCU1qKwFo4eleudO8OMGeUP8aqsunVlJduUFMmeo2PTVSBcvCjDlyrTU9i0qbR+p6Vp67cTOnWCX/5SPgdfYmMlCcZ11wW2XCp0aTChlHLMhg2QmSm9CzVJ9XrhAuzeDXPnyoJuwc5aWSDq3/+W1lzwb5ab6Gi48Ubp6ahXT3o+wm2yugou69bJfJ+rr67c8cOHQ8uWkj5WBc6FC/L9U9H3TefOkJQk17V3U1VEg4kAOnXqFCkpKaSkpNCiRQtatWp16fbFSjYf3nfffezatavcY1588UVH1lhYuXKlz0XwqqImr2/37t2keKamqITZs2eTkZFR7jEzZ87k5z//eZXOqyp27hysWQNXXCHDcmoiNlbmGrRpIy2j69YF7w9gQQEsXAjr18OgQdC6de08jzEyVAGk9+f11+U9V8rfsrNlDkTPnvI/WBmxsfDggzX/31dV8+mn8K9/Ve77MT8f5s+XRh+lyhOwResUJCYmkpaWBsDvfvc7GjVqxGOPPVbiGGst1lqiymg2ePXVVyt8np/+9Kc1L2w1rFy5kqSkJJ8L4VVWoF/f7Nmz6devHy1atPDbOVXlrF4tP1Zjx/rnfLGxskr0e+/JD+b58yUXvwsGubkyOfrAAemNGTIkMM/brZsEMLNmyXuUkBCY51WRYeVKmQNR1f839yKU33wja1LoOim1Kz8fvv1Wvg8q0xNcp44MX1u9WuZX6KKYqizaMxEEdu/eTa9evfjRj35Ev379OHr0KDNmzCA1NZWePXvyzDPPXDp22LBhpKWlUVBQQHx8PE888QR9+vRhyJAhHHclpn/qqaf43//930vHP/HEEwwcOJCuXbuyzpWP79y5c0yaNIk+ffpw5513kpqaeinQ8fT444/To0cPkpOT+fWvfw3AsWPHuOWWW0hNTWXgwIGsX7+ePXv2MHPmTP7yl7+QkpJy6XncnnrqKaZNm8bYsWNp3749ixcv5pe//CW9evViwoQJFBQUVOv1ecvPz+fuu++md+/e3H777eS68oY+/fTTDBgw4NL7bK1l/vz5pKWlcccdd1zqHdqwYQNDhgyhT58+DBo0iPPnzwOQnp7OuHHjuOKKK3jyySer90GrEhITYehQmfPgL3XqSJanK6+UVV2DzY4dkJ4uZQxUIAFSebj3XglmZs3ScerKf4qKZBjd4MFlj8Evz/798OGHsHGj34umvPhaW6I87lTceXkSUChVFg0mgsT27dt54IEH+Oabb2jVqhXPPvssmzZtYsuWLSxbtozt27eXekxWVhYjRoxgy5YtDBkyhNmzZ/s8t7WWr776ir/85S+XApN//OMftGjRgi1btvDEE0/wzTfflHrcsWPHWLp0Kd999x1bt269VIl+9NFH+dWvfsWmTZtYsGAB06dPp1OnTkyfPp3HH3+ctLQ0rvSRF3Dfvn0sXbqURYsWMWXKFMaPH8+3335LVFQUH3/8cbVfn/f7+NOf/pRt27YRGxvLyy+/DMDPfvYzNm7cyLZt28jKyuLjjz++FES4g4qioiImT57Miy++yJYtW/j000+pV68eAFu2bGHhwoVs3bqVOXPmcERrYzU2aFD5C9RVl/sHsEsXub1zp/NrURQWymXfvvCTn0CvXoEvQ5s2MociJgbmzZMWYaVqKioKJk2qfi9gp06yrV7t/P9puHOvLVGV1NPNm0uv0VdfycrmSvkS0cOcXnut9H09e8KAAdId6GvaQUqKbOfPS+YKT9OmVb8snTp1YsCAAZduz5s3j1mzZlFQUMCRI0fYvn07PXr0KPGY+vXrc+211wLQv39/Pv/8c5/nvuWWWy4ds3//fgDWrl17qaehT58+9OzZs9TjEhISiIqK4sEHH2TChAlMnDgRgOXLl5eY15CZmXmpB6A81113HXXq1KF3794AXHPNNQD07t37Urmq8/o8dejQ4dIwq7vuuot///vf/PznP2fFihX85S9/IS8vj5MnT9K/f/9L53bbsWMHbdu2pV+/fgA08UitM2bMGOLi4gDo1q0bBw8epGXLlhWWR5V24ACcOQPJyTWbdF0ZZ8/KHIr4eBne449sSVW1bx8sXgxTpsgPc2Ji4MvglpQE06fD6dPSi6NUTRw6JBP8L7us+v/L7uD/pZckoBg/3r9lVCI7W7LdXXVV1T+rq6+WxBarV0uWJ6W8ac9EkGjYsOGl6z/88AN///vfWblyJVu3bmX8+PHk+Uh5Ubdu3UvXo6OjLw0V8uZuXfc8xlZi9lVMTAybNm3ipptuYtGiRUyYMOHSY7/66ivS0tJIS0vj8OHD1K/EYEp3OaKiokqUPSoqymfZK3p9+/fvvzSBfebMmQAYr29JYwznz5/n4Ycf5r333mPr1q3cf//9Pt9Pa22px3uXvayyqMopKpI0sKtWFbfW16ZGjaQSn50tq0sfO1b7z+np229hzhxJ1xobG9jnLkujRsVDwL78UibBB+tkdRW8CgslSF64sOZ/P9r6XfsaNoTJk+V9rqpGjeSxXu1vSl0S0W1T5fUkxMSUv79Bg5r1RJQnOzubuLg4GjduzNGjR/nkk08Y7+fmmmHDhrFgwQKGDx/Otm3bfA6jysnJIS8vj4kTJzJo0KBLPSNjxozhxRdf5Be/+AUAaWlppKSkEBcXR05Ojl/LWZ727duXmOexe/du9u3bx8aNGxkwYADz5s1j2LBh5ObmEhUVRVJSEjk5OSxatIipU6cClChzz549OXDgAF9//TX9+vUjOzu7RJCnam7LFqnQ33Zb4FrGO3SQTE9z5sgq05Mny3217csvJf1ru3bynME2edFa+SzS0mRF3AkT/JueVoW3TZuk4j9lin96GK++WnosA9HIEImio6Fr1+o/3p11yx041navsgot+tMRhPr160ePHj3o1asXDz74IEOHDvX7czzyyCMcPnyY5ORk/vrXv9KrV68Sw3pA5ixMmDCBPn36MGrUKP72t78Bkpr1iy++IDk5mR49evDKK68AcOONN7JgwQL69u1bagJ2oPTs2ZNXXnmF5ORkzp07x4wZM0hMTOTee++lV69e3HzzzQwaNOjS8ffddx/Tp08nJSUFYwzz5s3jxz/+MX369GHs2LFcuHDBkdcRji5ehBUrZOy+14i9Wte8uQzvadwYDh+u/efbtk0CiR49ZJXuYAskQCoDN94o+f43b5YUkLoWhaqM3FzpXezYUVI7+0OjRvK/UpMV4JVvR45Ixq2azkk5e1bWxtmyxT/lUuHDVGa4SyhKTU21mzZtKnHfjh076N69u0MlCi4FBQUUFBQQGxvLDz/8wNixY/nhhx+oowOpa0T/xsr22Wcy5nb69NpbW6EiFy9Kr6MxMvSpcePaeZ7CQvj6a+jfPzRa+zdulOFnrVvDffeFRpmVcz75RNZJeegh8HdW7bNn5X9n+HBt/faX99+HrVvhscdkjkt1WSvDRXNy4OGHZfimihzGmM3W2lRf+7TmGKHOnj3L6NGjKSgowFrLyy+/rIGEqlWtWsGwYc4FElD843fmjEz47NtXJn/6o9KSlyeVrDFjZHyyRz6FoDdggLQM5+RoIKEqVreu/M3UxvI8e/dKK3qTJrK2gaoZ99oSPXrULJAA+Z4cN06Gi375JYwY4Z8yqtCntccIFR8fz+bNm50uhoogXboUp2t1WuPGUlH58kvpobj55prN4cjOljkZp07Jj7a/hn4EkmeH2t69MjTr8sudK48KXldfXXvn7t1bVlxesUL+JrX1u2aqurZERdq2le+4tWtlMrcryaGKcNoGpZSqVUeOyPjqYBqPHxUlKSjHjoXvvpNAwEeCr0o5fly6/rOyYOrU0AwkPBUVwccfw6uvwu7dTpdGBZP0dPjhh9p9Dnfrd3a2BPuqZrZskV6eqqwtUZExY+R7Yv16/51ThbaICybCdY6Icp7+bZVmrQz92bhRfnyCiTGyUvakSZIvf+3aqp8jPV26/K2VuQbujCehLCpKJsImJMBbb+lkSyWslXk1779f+wseerZ+BzBBYNgpKpKU1P37+3f+SUIC3HMPjBrlv3Oq0BZRw5xiY2M5deoUiYmJZa4noFR1WGs5deoUscGymECQ2LlTFqm7/vqaj9etLb17Q9OmxeO/ra38D2/TplLxue46WRgvXMTFSXA0fz689560Eg8bpq8PKtMAACAASURBVBNiI9nWrdLLeMstgUnrPGaMLqxYU1FRcOuttXPudu3kMj9fPif9bohsEfWv2rp1a9LT0zlx4oTTRVFhKDY2ltZOzi4OMoWFsGyZpHrs29fp0pTP/bHl5kpr/OjR5Q8L2LFD5n80bCh59sNRvXoybOs//5EVs1Xkys+XOQwtW0rwHQgJCbrack1YC5mZ8j7WlhMn4I03YOLEmq1hoUJfQIMJY8x44O9ANDDTWvus1/52wGygGXAauMtam+7a92dgAjI0axnwM1vFcSUxMTF0CMRqVUopvvpKKqF33RU6GYIuXpS5E2++KRWZnj1L7rdWAqR166Q3YuBAZ8oZKNHRMjnd3Vtz5owEUDExTpdMBZI7UcGkSYFvgT5+XNZtGTVKW7+r4vBhmcs1eTJ061Y7z5GQII0On34KnTvL94WKTAH7iTfGRAMvAtcCPYA7jTHeS1c9B7xhrU0GngH+6HrslcBQIBnoBQwANCmZUkGsfXvJFd+5s9MlqbwmTWS17FatYOHCkhMMCwpg0SIJJAYOhFSf2bbDjzESDBYWSpD1xhtw/rzTpVKB1Lix/L27h7YE0v798Pnn8P33gX/uUJaWJkF/bbafRkdLEotTp2RFdBW5AtleOBDYba3da629CLwN3Oh1TA9ghev6Zx77LRAL1AXqATHAsVovsVKq2i6/XIYLhZr69WUCcrduktVo/XrprZg7V/K1jxkD114bOr0t/hIdLZ/n0aMy6fzMGadLpAIlJUWGsjihf39ISpLW78JCZ8oQatxrS3TvXvtz1a64QhJPrFpV8xW2VegK5DCnVsAhj9vpwCCvY7YAk5ChUDcDccaYRGvtl8aYz4CjgAFesNbuCECZlVJVdPKkZGEZM0YWQgtFMTFw222wZo0MdcrKgowMGfITyQtp9eghn+lbb8kQijFjivPXf/ll6fS6SUnFY+zXri2dHrhFi+L1LVatkuFUnlq1krkpRUWyerq3tm2hUyc5r69sXB06SA9ZXp7vNKNXXCHzZc6elYxj3rp1k6A4K0tWZfbWs6fMCTp9ujjrVdOm8jcSDkNyjh2DgwedXcnd3fr91lvS+j3Iu9agStm1S/7m/bW2RHmMkc/n5ZdlOFq4D/1UvgUymPD11eo95+Ex4AVjzDRgDXAYKDDGdAa6A+7ZrcuMMVdZa9eUeAJjZgAzANq2bevHoiulKmvZMhmacM01TpekZqKiYORIuR4XBz/7maRZjHRt28IDD8Dbb0vlwV1h+eqr0r0V3boVBxPr1pVuuezTpziYWLu2dMvzgAESTFgrgZ23YcMkmCgo8L0/Oro4mPC1v0EDCSbOn/e9Pz5egonsbN/7mzeXYCIzU/a7g6EDB6QlP5THkFsrPXMZGdCrl/TYOcWz9Ts52dmyhIKtW2XIZqCmiLZoAQ89JP8PKjKZQOXGN8YMAX5nrR3nuv0kgLX2j2Uc3wjYaa1tbYx5HIi11v7ete+3QJ619s9lPV9qaqrdpIP4lAqoffvg9delxXrYMKdLo1RgWSsV3tWrJUi64w6nS1R9338vvQHXXhscvQEZGbLA5PDhuip2RXJzpbesVStnnluDvfBkjNlsrfU5WzCQHZcbgSuMMR2MMXWBycASzwOMMUnGGHeZnkQyOwEcBEYYY+oYY2KQydc6zEmpIFJUJAvUxcfD4MFOl0apwDMGrr5a1lVJTna6NNVXWChzFBITgyfRQIsWMmdHA4mK1a/vTCCxfz/87W/SM6ciS8CCCWttAfAw8AkSCCyw1n5njHnGGHOD67CRwC5jzPdAc+B/XPcvBPYA25B5FVuste8HquxKqYpt3Sqth7rYlIp0/fsXD9/67rvQW6dj82aZ+zR2bPAN1dqzR4ZSqtKslYxzu3Y58/ytWkkg88knpec/qfAW0J98a+1SYKnXfb/1uL4QCRy8H1cIPFTrBVRKVVunTtIq6702g1KR6uJF+OgjqVhNmeJMa3F1NGtWPF8l2KSnwxdfSNmcSFUbzA4flnlMTi2nFRMjvUfvvSflCOXeOVU1EZbcUClVW+LiYMSI8Mhio5Q/1K0L990nl6+9FjprJXToABMmBOf/8pVXyroX2vpdmnttCScbdJKTZaX05ctLZ29T4UuDCaVUjWRnw5w5MixCKVVSYqJkv2rWTDJg+UoxGywyM2UIkXeK32Dibv0+ckRav5UoKAjc2hLlMQbGjYOcHNi717lyqMDSYEIpVSMrV0oWp2AbW61UsGjUCKZNk6GA2dlOl6Zsy5ZJit9gb1HW1u/Sdu4M3NoSFWnXDh59FLp2dbokKlB0mqRSqtqOHpXFuq68UhbrUkr5Vrcu3Hln8dChU6fkfyZYVlI/eBC2b5d5T3FxTpemfMZIytrjx7URwy02Vnol2rd3uiTC/XuQlSVrXqjwpsGEUqparJVxy/XrS+53pVT53IFDbi7MmiUL5t16q/PpTt3/y40bw5Ahzpalstq0kU2Jzp1lCybbtslkbF3QrpqKiuDcueLNGFm9MQhpMKGUqpbvv5e84hMm6MrQSlVF/fowahR8+KEs8jhlCjRs6Fx5vv1WMgHddJPzgU1Vbd4sqXevucbpkjjn6FFZ3yfYFovr3Fnmb3zyCdx9d3BO6A+YgoLioODsWdl8Xfe8Lze3ZJaB9u3hyScdewnl0WBCKVUtHTvKUIP+/Z0uiVKhJzVVhhMtXCi9FHfdBQkJzpTl8stlock+fZx5/po4eRLWr4fevWVhu0hjLbz7rgSj06Y5XZqS6teHkSMlPfLu3XDFFU6XyE8uXiw7GHBf977tj6wG587V/By1RIMJpVS1xMTAoEFOl0Kp0NW1K9xzD8ybB2vXwg03VPyY2pCUBOPHO/PcNXXVVZIS9dNPI7P1+8gROHEieIenpabKpP5PPpEGKL/NcbFWlmovKJBL93Xv2zXZd+FCyaDAHRg4NetfgwmlVLjIzZVUsGPH6qJRStVUmzYwfXrxpOeiosBNyj57Virho0eH7iRZz9bvH34IzoX2apN7bYkePar4QGvh/HnJB5yZKRXkwkL5A6zJVlgo53Zdj7aWsTkNWbSxHcdyv6dlXE7F56hMpb+oqFbez6DToIF0OzVqJJu1QRkxazChlKqSNWukNUznSSjlH+7hTXl5Modi4EDo27f2n/ezz2S+xIgRtf9ctcnd+v3ppzJOP1gyZNW2ggKZ5Ny9u4/v4/x8CRJOny7ePG9nZkrLewB0sfCzpnVpuPtiQJ4vKBkjwUDDhiWDA1/XPS9D5I9ZgwmlVKWdOiU/2n37anYOpfzNGGmI/M9/ZD2Kq66qvUbIY8dkAb1Bg2RhvVAWHQ0TJwa2V8dRRUWQnc2Bb7LJ21uHlJZ7Yf7hkgFDTo7TpbzEGGgYcxFr4UReHJfVD56yVUudOqUr/r6ue96uXz8oexT8RYMJpVSlLV8uP9yjRjldEqXCT716ktlpyRLpNcjOlmxp/q4gWyut+LGxQdwrUVQkresFBXLpufm4r4P7vr3lH0dRkYwLqltX3vC6dYs379tl7YuJqb2oxVoZS+qrJ8HzelERnYBHLzQgfm0uGFvhqZ22NqMzq4524eGeq2ha73zNT1injvwgRUeXvu6PfXXr+u5JqFcvrAOD6tBgQilVKenpsGOHBBKNGjldGqXCU3S0pGht3Bg+/1zqM+PG+fc5du+GPXtk0rXf04nm5RVXfM+ckcusLMmAU1ZAUFalv4qsheWHuxNtihjVapefX5gXd0BSmeCjrEDl4sWSw5DcgUIVMv8kxNagUl63rqwul5AgkWV0tFSSo6MlWKrqVsHjU3LrsGZBC5a3T+a2cTllP74yFX5jtEIfRDSYUEpVSqtWssBW165Ol0Sp8GaMTIpOSIBOnfx//latpEdiwIAqPtBXoJCZWfK6P1JgVpMxcK6gLttOt6Jv0iH/tH6XxR30OJRhZ/2xDhw4m8itHTYTHeWjV8IYWXwiIaF4cwcO7q1Bg4BWyOOAoRZWrYLBTXXRwXCiwYRSqkLuBBK9ejldEqUih3sSdlGRDEsaONA/a1E0aABXX+11pztQ8Ny8gwYHA4XKGtVyF9+dbsnyw924rePXThenVlgLm7OvoH6TukT3TS4dJCQkSHquIJxAcuWVstDgxx9LFjPtXAgPGkwopcqVnw+vvCKTQTWYUCrwMjNhyxbJ3DN1KrRsWcUTuAKFvIwzLHwvmtGdD3A5RyVQcAcNwRYoGCNDidxbnTpl3/a43jgmhqGdkli1vQeDhnWjbavCksdFRcmX2sWLxduFCyVve2++9temmBjfAYLrviPnm3LijXpcfz0QYouG1q0rvW4rV8rot/h4p0uk/EGDCaVUudatg+PHZQy3UspPPBfdqmCCcWJ+Pg/0gTkfNuW138Dtw47QuVlW8XG+zpGfLxl9PAKFz9O7s+dYJ0Z3XwMNsmvnddWpI5Ve7y02tvzgwPu+qKhqN1tfeRE2/wOW53fh/tpIFmFtyYCkOsHIhQvyer2HHzVtKpPSynntaR/KQ3v2rIXXFgB9+kjZY2KcLonyFw0mlFJlysmRlXl79IC2bZ0ujVJBKDdXFl45fFguz5ypVPYhCgqkUlpJScADF+sxd/cg3toSx03t00hOPFzpx2deaMD64x3ok3iIy6sbSJQVKHhuFVSEA6FuXbj55lpsADGmeBJ1gBUUyNogPteWCBHuTqeCAsjIgNatnS6RqikNJpRSZVq5UsZrjxnjdEmUclh+vtR83IGDe8vMDFgR4upe4L6u63h3X98qTy5ecbgbUVhGtSwjy1GdOsUTdoM4UKisjh2LrwfposHVUlgoc2dqY2J+oH38MWzdCo8+qhkCQ50GE0opn86cgbQ0GDLEP5M+lQoJRUVw4kTJoOHIERnrV410pf5WL7qAOztvvHT7+zOX0bnJCaLKWWcgPS+Jb/M6MyL1FI0H9An5QKGyCgth4UKZYzJ8uNOl8Y969XxMng9RQ4bIwomrVsmigyp0aTChlPIpPh7uuw+aNXO6JErVAmtlBqhnL8ORI3D0qPRCBEJUVPnzBiq470h2I976tD1dm1/g1jFniImNLnlM/fqQkECL2EZc+7WR7FCBH5njmOho+Zg//1wyY4V66/fZs7LezxVXyGsLdYmJ0suyYYNcXnaZ0yVS1WVsFcZshpLU1FS7adMmp4uhVEgqLAyPHyulADh/vmTA4L4874d1CKKioHlzWbyhVSupEdWrV7ngwA+pO7/6Cj76SJ56yhRJ+6qKnToFL74owcT11ztdmpr54gtYtgweeUQq4uEgNxeef17+fu+6y+nSqPIYYzZba1N97Qtoz4QxZjzwdyAamGmtfdZrfztgNtAMOA3cZa1Nd+1rC8wE2gAWuM5auz9wpVcqMhQWwssvQ3IyDBvmdGmUqoKLF6VnwTNgOHxYxuz5Q2Ki1Hpatiy+bNFCggOHDBwIcXGwaBHMmiUVsqZNZV9BAbzxhvwfd+niWBEd5dn6PWhQ6LZ+WyvDTtu0CZ9AAqTzbMQISXuclxe6k8ojXcC+AY0x0cCLwDVAOrDRGLPEWrvd47DngDesta8bY0YBfwTudu17A/gfa+0yY0wjwPnBq0qFoc2bZXi4Dm9SPlkrzYlZWTIcqKio/K2wUB7jeb2wsOLHlXcu78fn5UkQceJElTIklSkurjhY8AwcgrSm07073HMPzJsHhw4VBxPr18PBg5qCc8QIWadj1Sq4/XanS1M9R47In3eo9674MnCgBHphNmUnogSyOWUgsNtauxfAGPM2cCPgGUz0AH7huv4ZsNh1bA+gjrV2GYC19mygCq1UJMnNlR/cDh0ityUzYlkruYCzsireCgqcLq1/1KtXMmBwb3FxTpesytq2leEv7mFOJ0/KXIGuXeX/OZLVry9BRPPmTpek+tLSQnttifK4R/udOyd/t+3aOVseVXWBDCZaAYc8bqcDg7yO2QJMQoZC3QzEGWMSgS7AGWPMu0AHYDnwhLW2sNZLrVQt80xb+NFHspaRpzZtoL9rldP335dGWU8dOsgiQEVFsGRJ6fNfcYX8AF28CEuXlt7fvbtUOM6fh7ffloBi3DhtJQobhYWQnV1xgJCdHRTZimpFdLQMR/IMGlq2lPEiYfSH7g4kDh+GV1+Vj/Oaa5wtU7BwB1S5ufLn4MASEdVmrXymoby2RGX85z/yOn/849CfLB9pAhlM+PrG9u6Pfgx4wRgzDVgDHAYKkHIOB/oCB4H5wDRgVoknMGYGMAOgra6wdcmHH8oXp/6oBJ/t22Us79Sp8hkdPFh6Tqjnj8f+/aUbhT0XZtq3r/RzuMfXFhX53n/55XJZWCh1ypEjpd6lglx+fuV6Ec6e9c/Qn1CRlFQyYHBPinZwXkOgJSZCt27yf5yU5HRpgoe10mCSny+T1UOlwmoMPPhg6YamcDNqlMz7cc/9Cae5IdVRUCDBVSj01AQsm5MxZgjwO2vtONftJwGstX8s4/hGwE5rbWtjzGDgWWvtSNe+u4HB1tqflvV8ms2p2N/+Bu3by7jRFSvghhvCu3UjVGzYIIv2tG4tP2z16ztdIhUUCgokCDhzpvjS87o7SPBHJqLqqlcPmjSRy6ioym3R0VIrio6W257Xa7JFR8vWrJlExvXqOfe+qKD3/ffwzjsSSIRKhbWoyC+Jv0JCejq89ZZcnzIlclfHzsuD+fOlgfHRR+Xr1mnBks1pI3CFMaYD0uMwGZjieYAxJgk4ba0tAp5EMju5H9vUGNPMWnsCGAVopFAJ587J6IUWLWRS7c6dkipv6tSSLdoqcKyF5cslzV+3bjBpkk6QjAhFRcXDjbwDBc/bZx2cEtaggfxqVbRpa4QKUV26wL33SoV11qzgr7Dm5Eh2veuvl+Go4a51a3jgAZgzB1auhLvvDquRiJWSnQ1z58qE+xtvDI5AoiIBCyastQXGmIeBT5DUsLOttd8ZY54BNllrlwAjgT8aYywyzOmnrscWGmMeA1YYYwywGXglUGUPZceOyWWLFtCxowQRCxbIl+jUqaGbJi+UrVghgcSAAXDttZHT4hS2rJWo3bv3IDOzdG+CU8ON4uIqFyRoVKsigGeFdckS+NGPgvd7eNs2aV8IhR4Uf0lMlM/H3YEZST0zJ07I32VurtTROnVyukSVE9BBpNbapcBSr/t+63F9IbCwjMcuA5JrtYBhKCNDLt1j4Dt1klWN58yB2bNh2jQdHx9o/fpBw4YweHDktbiEpKwsSTtaVk+CU9mNoqKke7GiAKFxY12BUCkviYkwfbrMQ4iKKpkII1i415Zo3Try5r6457MUFkorfZs2Mp8v2D4jf9u+XV7zffcVz2cMBZEzIy1CxcZKNh/PVVFbtJAv0VWrIqu1w0k5ObJ+w4gRkJAAQ4Y4XSLlU1YWHDggA1UPHJAtKyuwZTBGAoD4eAkG3JdNmxYHCPHx8msb7r+sStWihg1ls1Yy5cXFBVeF9ehRGZ48caLTJXFWkyawerUM/5k4MTzbRi5ckOleV10l2RtDJTmAmwYTYa5fP9m8xcfDTTfJ9bw82LVL0osq//PstuzVK/JamIKWd+Bw8KD/VkouS8OG8s/n3tyBgef1xo0jp09fqSBgrWzBVmF1ry3Rq5fTJXFOdLQkjWncWD6fs2fhtttCK7VvRTZskDVhpk8vbicKNRpMhDH38OyKWlk2bIDPPpMWkDFjgqdVJhwcPCir0kZHS7elBhIOyc6WvLq1FTjExpbsPfAOFNxBgs5JUCroREUFZ4U1OVkW2ov0fAfGwNVXS8/Rhx/Cu+/C5MlOl6rmvJOxNGzodImqT4OJMHb0KLz+uvzTlbcC6vDh8uX5xRdS57rppuBolQl1O3fCwoVSp7zrLqlnqgDIzi4eouSPwCEmRtYqSEoqe9hRpP/aKxXivCus8+bBPfc427jWunVwZ5oKtNRU+Xzi450uSc0VFsLixTLBPhySsWgwEcYyMmQcXkVpxaKi4LrrpFVmxQpJTHP77Vo/qqm6daUOescdJeesKD+qjcChdWtZJahtW7m8/HKNrpWKEO4Kq3tZFKds3CiTjjVBSknu9LjWyrzPLl3kdzbUfP65BBKjR8OwYaE/IkSDiTCWkSETeirTIm6M9FA0biz/oBcvajBRHdbCoUNSD+3YUXqEQv1LImh4Bg7u4Uo1CRzq1JFfaw0clFIePNdz+PprGWoUyArr2bPw0Udw5ZUaTJQlLw+2boV162RIWpcuTpeoatyfbbduTpfEPzSYCGMZGfIlWJXKbJ8+0LOn1LPca2yFQ5diIBQWwn/+I19wM2ZAy5YaSFRLUZGsrJiR4d/AoXVrWQpeAwelVCUUFMDatZKNL5AV1q1b5WswJSUwzxeK6teXtSjmzoW335ZJ876SzQSTEydk9MfNN0tDb7gEEqDBRNiyVhasq06Gpjquv4pVq6Sr9c47pf6lynbhAsyfD3v3wqhRoZUf2hGFhRIwHD8u37DHjsnl8eNw8qT8klaXO3Bo165408BBKVVFderA/fcHtsIayWtLVFWjRrJW1jvvyOKD587JCItg5JmMJTsbmjVzukT+pcFEmCoogEGDpB5VXf36yQIqb7wBkyZB9+7+K184ycmRH5vjx2XyurYmuXgGDJ7biRM1DxjcvAOHtm2lS0gDB6WUH3hXWHNyZL2g2pKRoWtLVEW9etLg+cEHwRt87dgBixaFdzIWDSbCVEyMtJDXRHy8tMrMmwcLFki2gYED/VO+cPLDD3D6NEyZAp07O12aACsslMDAu3fh+HEJJPwRMLhp4KCUcoC7wvr++7Wf3fnUKQlgInltiaqKjoYbbyy+vWePTIdzOrUvyJC1996TOTdTpoRvMhZj3YsRhJnU1FS7adMmp4vhmKwsmUBdr17Nz5WfLylO9+yBRx6pODtUpLh4sfjLKjtbJq+HpYIC3z0Mx49LFOXPgMGtcWOZ8HP55cXzHDRwUEo5yHPtpqNHITGxdiqsRUWhnSbUSTk58Pe/y8/HlCnOr91w5gysWSONsaG+zJAxZrO1NtXnPg0mwtNbb8kf8U9+4p/zFRVJw7N7LoC1kT25eOdOaaW6++4wyLZhrXwDZ2bKH83JkyUDhlOnin9F/alJE7jsMtmaNZNv/2bN5LY/omCllKoFeXlSYU1I8G+F9cIFCU4i+bfVH9xrPDVuLMOKEhIC+/yFhfDNN9C/f3h9luUFEzrMKUxlZEiDrr9ERRUHEl9/Dd99J2tRRGKdb+NGWLpUui3j4pwuTQUKCiRAcAcK7kv3lpkp3ViFhbXz/PHxxQGC59asWWT+8SilQl5srAyrWbgQZs3yX4X13XdlJMA999T8XJGsWze4914Zoj1rFkydKh3bgXDhggwL37NH/iY6dgzM8zpNg4kwdP68DLuprRbz6GjYtw9efVX+SYO+Qu0n1sLKlbLYTNeucOutDnZbWgu5uWUHCu7rZ8/Wflni4333LmjAoJQKU+4K61tv+afCevaszL+78kr/lTGStWkjcz7nzIHduwMTTOTkyN/DsWMSbEZKIAEaTISljAy5rK1gok8f6dZdsKC4VSZYsyj40zffSCDRvz9MmFCLY1rdC3xUFChcvFhLBfDBHTB4b0lJGjAopSJSmzay1sGcObBpE9xwQ/XPtW2bfPVXJ5278i0pCR56qHgB3vPna28C9MmT8ndw/rxM1r/iitp5nmClwUQYcgcTzZvX3nN07izp8ubOlYDi4Yedn+hU2/r0kYRCvXv7aRxkQYEsTLFrFxw+XBwkZGfXzqTm8jRoIAFD06ayeQ9JCoa0GEopFWSSkuDBB4srrPn5Ve+xtlYaq1q3Dr/1B5xWv75cZmbCK6/A4MGyFoW/5zKcOyef47RpgRtSFUw0mAhDXbrIP1BtV+5btoTp06ULMVwDibNnZX7EhAnyGpOTa3Aya+HQIUk6vXOn9Gnn5/utrD4ZIxOd3YFCfHzp6/Hx2ruglFLV5P79y82FmTOl4akqFVZdW6L2NW4svQUrV0p73XXX+Wd0wenTMjeiXTvJdlknQmvVEfqyw1tSUuCGHTVtCgMGyPVDh2SsYKrPuf6hx91tee6cLJ9Q5YDJWnmgO3jYtUtO5i8xMcVBQVmBQpMmmmNQKaUCoG5d6V2oaoU1KUnm4HXqVPtljFTR0bKobOPGMlw5J6fm8x43bZLGxjvukHmUkRpIgAYTYaegQOqs7drJwjeBtHkzpKVJcqBRo0I7JVp6ukykMka6LVu1quQDs7IkcNi5U4KIzMzqFaBhw7IDBPf1Bg1C+01WSqkwUt0Ka0yMLlIXCMbA6NGSNOajj+Qzqs7ivtbCZ5/J+hFdukCHDv4va6jRYCLMHD8O77wjaVt79Ajsc99wg3yZfv65tMq4b4eaffskkIiLq0TKv9xc+P774t6Ho0er9mTx8ZIWpEsXGSzrDhRCfXUbpZSKQN4V1k8+KX/40t69MmVu8GD92g+UgQOlN6hNm6o/trBQ1phKSwtAMpYQosFEmKntTE7liYqSL83GjSVqP3sWJk8OvS/Iyy6T+v348T6GNuXnSwJpd+/D/v1VW9CtQQMJHLp3lydp3lx7F5RSKswMHCjtQhX1am/YAEeOwNChgSmXEu60rXl50gA7enTlJk7v3i2BxNVXw1VX6c+3mwYTYSYjQ8ZtNm3qzPMbAyNGSECxZ0/o9ExYK18QyckSQEya5NpRVAQHDxYPW9qzp2qTpmNiJPVVt26ytW2rzRhKKRUBunSRy8JCWLJEeh/ci79C8doSQ4boz4JTzp+X+ZGvvSYjOjp39n2ctVK/6doVZsyIzIxN5QloMGGMGQ/8HYgGZlprn/Xa3w6YDTQDTgN3WWvTPfY3BnYA71lrHw5YwUNIRob0SjgdLfftCykpUo7sbFkSIVjXoigshA8+kNR8WEvflseKg4fvv5dvm8oyRpYedwcPnTqFXteMUkopvzl7Vjqxd+woWWF1ry2RkuJo8SJaQoJkpZw7V4Y333hjq/OSOgAAIABJREFU6bU+Tp2C+fNlX6tWGkj4ErBgwhgTDbwIXAOkAxuNMUustds9DnsOeMNa+7oxZhTwR+Buj/2/B1YHqsyhxlrJphQsi964A5rFiyXIufPO6o1RrE0XL8KCV8+x+6vTjEz6lpS3V0HWmaqd5PLLJXDo3l1yz9XWqjhKKaVCTpMmJSusN9wgv9NpaVI51bUlnBUXB/fdJwHDe+9JA2O/frLPMxmLKlsgeyYGAruttXsBjDFvAzcCnsFED+AXruufAYvdO4wx/YHmwMdAmCQf9b8f/9jpEpQ2caKkWH39dcls0a2bwwWyFnbu5Oz6b3lrUV0yMgw3tNtKvzqHKvf4pk2L5zx07SoDY5VSSqkyeFZYFy+WTE+NGwfB76ECZKmlqVNh+fLinqNdu2DhwkomY4lwgQwmWgGetbV0YJDXMVuASchQqJuBOGNMIpAJ/BXppRhd+0UNTcYEZ702IQEeeECi+/nzpfF+yhTZt3q1dP96athQgg6Qf+zDh0vuj4+X7kaAjz+W3hhPSUmSYQEk68Lp064d58/BD7u5/MhmxjbZwLnzceScGsTkTlvpEn+87BfQsKEEDe4AolkzbaZQSilVJe4K69KlUmEdPtzpEilP0dEwbpxcz8uTXopmzeQzC9eFef0lkMGEr9qXdxqcx4AXjDHTgDXAYaAA+Amw1Fp7yJRTiTPGzABmALRt29YPRQ4tO3ZIa8fAgU6XpLSGDeHeeyVNXlZW8f1FRdKl6MnzdkX7CwtL7y8q8tifX0jh/iMyyy09HaylqOFZaALNG+TwaK+VxEQVlTxB3boS8bjnPbRpo8GDUkqpGouOhuuvd7oUqiInT0LPnhJc1K3rdGmCn7FVSWtZkycyZgjwO2vtONftJwGstX8s4/hGwE5rbWtjzFxgOFAENALqAv+01j5R1vOlpqbaTZs2+flVBLd582SNtJ/8xOmSBIHjx2HdOvjiC5kBXp6oKFl1xh08dOwY2UtZKqWUUkp5MMZsttb6nGZQ6RqTMeZ/kQxM31azHBuBK4wxHZAeh8nAFK/nSAJOW2uLgCeRzE5Ya6d6HDMNSC0vkIhUGRmy8nXEys+XlExr18pgx/LUqSMrzqSmSv6+2NjAlFEppZRSKoxUpfl1APCIMWYzMBN421pbQZNvMWttgTHmYeATJDXsbGvtd8aYZ4BN1tolwEjgj8YYiwxz+mkVyhfRzp+X4UNOLFbnuMOHJYBYv77iNK6tWslA1YEDdRCkUkoppVQNVWmYkzGmK3A/cBfQBHgXmGWtDbp0rZE2zGnfPsmWdM89xSs7hrW8PNi0CT7/vPQMbm/16knwMGyYdN3o/AellFJKqUrzyzAnAGvtLuDXrvkO1yGBxafGmIPALODf1trT5Z1D1Y7MTKkjN2/udElqkbUSNa1dK4HEhQvlH9+xowQQqakSUCillFJKKb+q7izTGKAx0jsRDRxE0rY+ZYyZYa19y0/lU5XUrx/07h2miy2fOydDmNauhSNHyj+2YUMYPFiCCF2mUimllFKqVlUpmDDGpCK9EZOB88DrwHRr7T7X/p8B/xfQYMIBYRVIWCuTqNeulUnVBQXlH9+9uwQQKSmaiUkppZRSKkCqks1pG9AVmUA9DfjQWuuV4Z+3kGBCBVBBgSwGN2hQ8cqNIevMGfjySwkiTp4s/9j4eLjyShg6VFaqU0oppZRSAVWVJtwFSAamw2UdYK09AUTVuFSqSk6ckDXZ+vZ1uiTVVFQE334rAcS2bSVXnfMWFSXjuYYNg1695LZSSimllHJEVYKJP+EjUDDGxAJF1tqLfiuVqpKMDLkMucnXJ09KALFuXcllsX1JSpIAYsgQ6ZFQSimllFKOq0ow8Q6wGvib1/0/QtaHuMlPZVJVlJEhy70nJDhdkkrIz4e0NAkidu4s/9g6daS7ZfhwWVhOU7oqpZRSSgWVqgQTQ4H/8nH/MuD/+Kc4qjoyMqRXIqjr2llZ8NlnsGaNZGcqT8uWEkAMGqQLyymllFJKBbGqBBMNAF8pdYqAOP8UR1VH/fpBPP/4yBFYtgw2bIBC7/n6HurVgwEDJIjQheWUUkoppUJCVYKJrcCdwNNe908BvvVbiVSVTZ7sdAm8uNO6LlsmE6vL06GDBBD9+0NsbGDKp5RSSiml/KIqwcTvgcXGmM7AStd9o4HbgJv9XTAVggoL4euv4dNP4eDBso9r0EAmUg8dCq1aBa58SimllFLKryodTFhrPzTGXA88BTzvuvsb4AZr7Ue1UThVsdWrYccOmDHDwSypeXkyoXrFCjh9uuzjmjWDMWMkkKhXL3DlU0oppZRStaJKSwVbaz8GPq6lsqhqOHJEOgQcCSTOnIGVK2VSdW5u2cd16ABjx8rq1LouhFJKKaVU2KhSMKGCT0YGtG0b4Cc9fFjmQ3z1VdmTqo2BPn0kiOjYUSdUK6WUUkqFoUoHE8aYukhq2DuBtkCM535rbbR/i6YqkpsrGVdbtAjAk1kr60IsWwbffVf2cTExMoxpzJgQXEVPKaWUUkpVRVUnYN8B/BH4v8DjQHtgMvAbv5dMVci98nWtBhOFhbBpkwQRhw6VfVzDhnD11TByJMRppmCllFJKqUhQlWDiduBH1tqPjTHPAf+x1u4xxuwArgFerpUSqjLVqwfJybUUTOTlweefy6TqzMyyj7vsMrjmGhg8WJbhVkoppZRSEaMqwURzYLvr+lkg3nX9Y+BP/iyUqpyWLeGWW/x80szM4knVeXllH9epk8yHSE7WSdVKKaWUUhGqKsHEQaCl63I3MA7YDAwByknlo2rLuXOyZINf5janpxdPqi4q8n2MMZKR6ZprJJhQSimllFIRrSrBxHvIInXrgb8D84wxDwKtgL/UQtlUOQoK4K9/hauukmkK1WKtLFLx6adyWZaYGFlgbvRoGdaklFJKKaUUVVu07kmP6wuNMYeAocD31toPaqNwqmwnTkgHQrNm1XhwQUHxpOr09LKPi4uTSGXkSGjUqJolVUoppZRS4apSwYQxJgaYA/wfa+0eAGvtBmBDLZZNlaNamZxyc4snVZ85U/ZxzZsXT6qOiSn7OKWUUkopFdEqFUxYa/ONMWOBJys8uBzGmPHIEKloYKa19lmv/e2A2UAz4DRwl7U23RiTAvwLaAwUAv9jrZ1fk7KEuowMSZ6UkFCJgzMzYflyWLu2/EnVnTsXT6rWReaUUkoppVQFqjJn4l3gFuC56jyRMSYaeBFJI5sObDTGLLHWbvc47DngDWvt68aYUciaFncD54F7rLU/GGNaApuNMZ9Ya8tpXg9vGRnSgVBunb+oCD76CD78sPyVqvv2lZ6Ijh1rpaxKKaWUUio8VTWb01PGmOHAJuCc505r7d8qePxAYLe1di+AMeZt4EaK080C9AB+4br+GbDYde7vPZ7niDHmONJ7EbHBxKBBFQQSx4/D7Nmwb5/v/e5J1WPGVHPihVJKKaWUinRVCSamAZlAsmvzZIGKgolWgOcSyunAIK9jtgCTkKFQNwNxxphEa+0p9wHGmIFAXWBPFcoednr0KGOHtTIv4p134OLF0vvj4mDUKBgxQlatVkoppZRSqpqqks2pQw2fy1c7uvW6/RjwgjFmGrAGOAwUXDqBMZcDbwL3Wvv/2rvzeDmrMsHjvyeQBAggZAUSElBQTHeDQgQVFQRBQJvFljaADggjaosOLYxb08pgCw7t0jItKj0ii8o60KIGMGPjwjQiF0ICAQJhTUgquTEsiSFAkmf+eN9IWamb3KW2e/P7fj73U1XnnLfquYeXyvvcc857coPNECLidOB0gMmTJw8w3M717LOwalWx+PrP9ot77jm44gq4//4ND9puOzjmGBdVS5IkqWH6MjIxUAuBXateTwIWVTfIzEUU6zKIiG2Bv8nM58rX2wM/B87JzN/V+4DMvAS4BGDatGm1icqQMWtWsUH1F75QlUzccw/88IfFTna19tkHPvShIqGQJEmSGqTXyUREXLSx+sz81Cbe4i5gz4jYnWLEYTpwYs1njAWWl6MOn6e4sxMRMYJi07wrMvO63sY8VFUqMHZsOcDwwgtw9dXwuzr51ciR8IEPwFvf6t2ZJEmS1HB9GZn4q5rXw4G9yve4Z1MHZ+aaiDgDuJXi1rCXZubciDgP6MrMm4CDgQsiIimmOX2iPPxvgXcAY8opUACnZOa9fYh/yKhUYPJk4OGH4Qc/gOXLN2y0xx7w4Q8XWYckSZLUBH1ZM/HO2rKI2Ar4PvDbXr7HDGBGTdkXq55fD1xf57gfUmyat9l74QV4bvladvrjb2DmNcWC62pbbFGsjTjssJoFFZIkSVJjDWjNRGaujoivUIw2fLcxIWljKrMWw2/nsdPO/xe2r0kkdtkFTj0Vdt21/sGSJElSAzViAfY4YNsGvI82Zt06mDmTiTf8jJN33o5dRj33Sl0EHHooHHusd2qSJElSy/RlAfana4uAnYGTqJm6pAZbtqxYGzF/PiOA3bf/wyt1O+5YrI143evaFp4kSZI2T30Zmfhkzet1QDfwA+CChkWkV2TCf/4nXHMNvPgiAHctncLO2zzHpG2fLfaMmD4dtt66zYFKkiRpc9TKTevUFytWwJVXwuzZfypas24YNy/4S9465WkmnXUA7LtvGwOUJEnS5q4v05xGAMMyc3VN+VbAusx8qdHBbbbmzCl2sl6x4s+Kl63elnXjJrDT2e+Efd2ATpIkSe3Vl2lO1wG/Br5RU/4xiv0hjm1QTJuv1avhuuvg9ts3rBsxgsr+0+GpfdlpTzegkyRJUvv1JZk4EPiHOuUzgS80JpzN2KOPwqWXFouta+22G5x6KpVZExhegdGjWx6dJEmStIG+JBPbAGvqlK8DnHPTX2vWwM9+BrfcsuEGdMOGwXvfC0ceCcOG0d0NEya4F50kSZI6Q1+SiTnACcCXaspPBO5vWESbk0WLitGIBQs2rJswodiAbrfd/lR00knFDtiSJElSJ+hLMvFl4N8jYg/gP8qyQ4HjgeMaHdiQlgm//CXceGMxMlHrne+E970PRoz4s+Jhw2DUqBbFKEmSJG1CX24N+/OI+GvgHOCisngWcHRm3tyM4Iak5cvhsstg3rwN63bYAU4+GaZO3aDqySfhvvuKPMOEQpIkSZ2gLyMTZOYtwC1NimVoy4Tf/x6uuqr+XKVp0+DEE3vMFB5/HO6+G9797ibHKUmSJPVSX/aZOAggM39dpzwz8zcNjm3o+OMf4Uc/KrKBWltvXSyGeNObNvoWlQqMGQPDhzcpRkmSJKmP+jIy8U3gvDrl2wPnAvs1IqAhZ+5cuPxyeO65Dev22gtOOQV23HGTb1OpwKRJjQ9PkiRJ6q++JBOvA2bXKb+vrFO1F1+EG26AX/1qw7rhw+G44+CQQyA2vQHdCy/As88WM6EkSZKkTtGXZOIFYBfg8ZryScBLDYtoKHj88eKWr0uXblg3eXJxy9edd+71261YUazN3mmnBsYoSZIkDVBfkolbga9GxNGZ+QxARIwGzi/rBEUCceGFsG7dn5dHFJvPvec9sGWf1r0zfjyceWYDY5QkSZIaoC9XtWcDvwGeiIg5ZdneQDcwvdGBDVrjx8Nb3wq33/5K2bhxxWjEq1/dvrgkSZKkBuvLPhOLI2If4CTgDUAAlwM/zsxVTYpvcDr+eHjoIVi2DN7+9uL1yJH9frsf/xgmToSDDmpgjJIkSdIA9W2+TbE2Yi6wAli/PfP7I4LMvKKhkQ1mW20Fp50GK1fC3nsP6K3WroVHHy0GPCRJkqRO0pd9JvYCfgrsTjEqsbY8/mXgRcBkolqDpjR1dxcJhYuvJUmS1GmG9aHtvwB3A68CVgGvB6YB9wJ/05s3iIgjImJeRMyPiM/VqZ8SEb+MiDkR8auImFRVd3JEPFL+nNyHuAe1SqV4NJmQJElSp+lLMvEm4J8y84/AOmDLzLwH+Azw9U0dHBFbAN8GjgSmAidExNSaZl8DrsjMvSk2yLugPHY08CXgAGB/4EsRsemd3oaASqXYlmL06HZHIkmSJP25viQTQTEiAcUdnCaWzxcCe/Ti+P2B+Zn5WGa+BFwNHFPTZirwy/L5bVX17wZmZuby8ra0M4Ej+hD7oLXDDsWyi2F9+S8lSZIktUBfFmDfD+wDPAb8HvhsRKwFPgLM78XxE4EFVa8XUow0VJtNMWXqW8BxwHYRMaaHYyeyGXjzm9sdgSRJklRfX/7e/RWK0QmAc4BdKUYPDgc+1Yvjo05Z1rw+GzgoImYBBwFPA2t6eSwRcXpEdEVEV3d3dy9C6mxr1264950kSZLUKXqdTGTmrZl5Q/n8scycCowFJmTmr3rxFgspEpD1JgGLaj5jUWa+LzPfCPxDWfZcb44t216SmdMyc9q4ceN6+6t1rEcegfPPhyVL2h2JJEmStKEBzcQv1zBsMELQg7uAPSNi94gYQbFr9k3VDSJibESsj+nzwKXl81uBwyNix3Lh9eFl2ZBWqRSjEztuFkvNJUmSNNi0bFlvZq4BzqBIAh4Ers3MuRFxXkQcXTY7GJgXEQ8DEyimVpGZy4EvUyQkdwHnlWVDWqUCY8bAiBGbbitJkiS1Wl93wB6QzJwBzKgp+2LV8+uB63s49lJeGanYLFQqMGnSpttJkiRJ7eANRzvU6tXw7LNuVidJkqTOZTLRoTLhkEPgNa9pdySSJElSfS2d5qTe23preMc72h2FJEmS1DNHJjpUdzesWrXpdpIkSVK7mEx0qBtuKH4kSZKkTmUy0YHWroWlS118LUmSpM5mMtGBli0rEgqTCUmSJHUyk4kOVKkUjyYTkiRJ6mQmEx2oUoHhw2H06HZHIkmSJPXMW8N2oP32gylTYJipniRJkjqYyUQHGju2+JEkSZI6mX/77jCrVsHs2e4xIUmSpM5nMtFhFiyAG2+E5cvbHYkkSZK0cSYTHaZSgQgYP77dkUiSJEkbZzLRYSoVGDMGRoxodySSJEnSxplMdJhKBSZMaHcUkiRJ0qaZTHSQ1avhmWfcrE6SJEmDg7eG7SAjR8JZZ7m/hCRJkgYHk4kOEgHbbdfuKCRJkqTe8W/gHeTuu6Grq91RSJIkSb1jMtFBurrgwQfbHYUkSZLUOyYTHWLtWli61MXXkiRJGjxamkxExBERMS8i5kfE5+rUT46I2yJiVkTMiYijyvLhEXF5RNwXEQ9GxOdbGXcrLFtWJBQmE5IkSRosWpZMRMQWwLeBI4GpwAkRMbWm2TnAtZn5RmA6cHFZfjwwMjP/CtgP+GhE7NaKuFulUikeTSYkSZI0WLRyZGJ/YH5mPpaZLwFXA8fUtElg+/L5q4BFVeWjImJLYGvgJeD55ofcOitXFreGHTOm3ZFIkiRJvdPKW8NOBBZUvV4IHFDT5lzgFxHxSWAU8K6y/HqKxGMxsA3w95m5vKnRttiBB8Jb3uIeE5IkSRo8WnnpGnXKsub1CcBlmTkJOAq4MiKGUYxqrAV2AXYHzoqIV2/wARGnR0RXRHR1d3c3NvoWMJGQJEnSYNLKy9eFwK5VryfxyjSm9U4DrgXIzDuArYCxwInALZn5cmYuBf4fMK32AzLzksyclpnTxo0b14RfoTmefx4uuwyeeqrdkUiSJEm918pk4i5gz4jYPSJGUCywvqmmzVPAoQAR8XqKZKK7LD8kCqOANwMPtSzyJlu8GJ54otgBW5IkSRosWpZMZOYa4AzgVuBBirs2zY2I8yLi6LLZWcBHImI2cBVwSmYmxV2gtgXup0hKfpCZc1oVe7NVKkUiMWFCuyORJEmSeq+VC7DJzBnAjJqyL1Y9fwA4sM5xKyluDzskVSowejSMGNHuSCRJkqTec8lvB6hU3F9CkiRJg4/JRJutWwdjx8KUKe2ORJIkSeqblk5z0oaGDYOTTmp3FJIkSVLfOTLRZlm704YkSZI0SJhMtNlPfwqXXtruKCRJkqS+M5los8WLYfjwdkchSZIk9Z3JRButXQtLl3onJ0mSJA1OJhNttGxZkVCYTEiSJGkwMploo0qleDSZkCRJ0mBkMtFGO+wA++4LY8a0OxJJkiSp79xnoo2mTHGzOkmSJA1ejky0SSY884z7TEiSJGnwMplok+efh299C7q62h2JJEmS1D8mE23i4mtJkiQNdiYTbVKpQASMH9/uSCRJkqT+MZlokyVLYPRoGDmy3ZFIkiRJ/WMy0SaVilOcJEmSNLh5a9g2ede7YJtt2h2FJEmS1H8mE20ydWq7I5AkSZIGxmlObbBkCTz1lHtMSJIkaXAzmWiDO++Eq69udxSSJEnSwJhMtMH6xdcR7Y5EkiRJ6r+WJhMRcUREzIuI+RHxuTr1kyPitoiYFRFzIuKoqrq9I+KOiJgbEfdFxFatjL1R1q6FpUu9k5MkSZIGv5YtwI6ILYBvA4cBC4G7IuKmzHygqtk5wLWZ+Z2ImArMAHaLiC2BHwIfyszZETEGeLlVsTfSH/4Aa9aYTEiSJGnwa+XIxP7A/Mx8LDNfAq4Gjqlpk8D25fNXAYvK54cDczJzNkBm/iEz17Yg5oarVIpHkwlJkiQNdq1MJiYCC6peLyzLqp0LfDAiFlKMSnyyLH8tkBFxa0TcExGfaXawzbLXXnDqqTB2bLsjkSRJkgamlclEveXGtTdHPQG4LDMnAUcBV0bEMIrpWG8DTiofj4uIQzf4gIjTI6IrIrq6u7sbG32DjBgBkyfDMJe+S5IkaZBr5SXtQmDXqteTeGUa03qnAdcCZOYdwFbA2PLYX2fmssxcRTFqsW/tB2TmJZk5LTOnjRs3rgm/wsBkwq9/DYtqf2tJkiRpEGplMnEXsGdE7B4RI4DpwE01bZ4CDgWIiNdTJBPdwK3A3hGxTbkY+yDgAQaZFSvgtttg4cJ2RyJJkiQNXMvu5pSZayLiDIrEYAvg0sycGxHnAV2ZeRNwFvBvEfH3FFOgTsnMBJ6JiG9QJCQJzMjMn7cq9kZx8bUkSZKGkpYlEwCZOYNiilJ12Rernj8AHNjDsT+kuD3soLU+mZgwob1xSJIkSY3gMuAWqlRg9GgYObLdkUiSJEkDZzLRQsuXO8VJkiRJQ0dLpzlt7j76UXjppXZHIUmSJDWGIxMtFOEUJ0mSJA0dJhMtMncu/OQn8PLL7Y5EkiRJagyTiRZ59FGYNw+2dGKZJEmShgiTiRapVIrF1xHtjkSSJElqDJOJFli3DpYu9U5OkiRJGlpMJlpg2TJYs8ZkQpIkSUOLyUQLrF4NY8eaTEiSJGlocTlwC0yeDGec0e4oJEmSpMZyZEKSJElSv5hMNFkmXHwx3HFHuyORJEmSGstkoslWrCju5LTFFu2ORJIkSWosk4kmq1SKRxdfS5IkaagxmWiy9cnEhAntjUOSJElqNJOJJqtUYPRoGDmy3ZFIkiRJjeWtYZtsl11g/Ph2RyFJkiQ1nslEk73tbe2OQJIkSWoOpzk10csvw9q17Y5CkiRJag6TiSaaNQvOPx9Wrmx3JJIkSVLjmUw0UaVSLLweNardkUiSJEmN19JkIiKOiIh5ETE/Ij5Xp35yRNwWEbMiYk5EHFWnfmVEnN26qPuvUin2l4hodySSJElS47UsmYiILYBvA0cCU4ETImJqTbNzgGsz843AdODimvpvAjc3O9ZGWLeu2PnazeokSZI0VLVyZGJ/YH5mPpaZLwFXA8fUtElg+/L5q4BF6ysi4ljgMWBuC2IdsGXLYM0akwlJkiQNXa1MJiYCC6peLyzLqp0LfDAiFgIzgE8CRMQo4LPA/9jYB0TE6RHRFRFd3d3djYq7X7beGg47DCZPbmsYkiRJUtO0Mpmot3Iga16fAFyWmZOAo4ArI2IYRRLxzczc6H2RMvOSzJyWmdPGjRvXkKD7a7vt4MADYYcd2hqGJEmS1DSt3LRuIbBr1etJVE1jKp0GHAGQmXdExFbAWOAA4P0RcSGwA7AuIlZn5r82P2xJkiRJ9bQymbgL2DMidgeeplhgfWJNm6eAQ4HLIuL1wFZAd2a+fX2DiDgXWGkiIUmSJLVXy6Y5ZeYa4AzgVuBBirs2zY2I8yLi6LLZWcBHImI2cBVwSmbWToWSJEmS1AFiqF6rT5s2Lbu6utodhiRJkjSoRcTdmTmtXp07YEuSJEnqF5MJSZIkSf1iMiFJkiSpX0wmJEmSJPWLyYQkSZKkfjGZkCRJktQvJhOSJEmS+mXI7jMREd3Ak20OYyywrM0xDFX2bfPYt81j3zaPfdsc9mvz2LfNY9823pTMHFevYsgmE50gIrp62uBDA2PfNo992zz2bfPYt81hvzaPfds89m1rOc1JkiRJUr+YTEiSJEnqF5OJ5rqk3QEMYfZt89i3zWPfNo992xz2a/PYt81j37aQayYkSZIk9YsjE5IkSZL6xWSiASLiiIiYFxHzI+JzdepHRsQ1Zf2dEbFb66McfCJi14i4LSIejIi5EfHf6rQ5OCKei4h7y58vtiPWwSginoiI+8p+66pTHxFxUXnezomIfdsR52ATEa+rOh/vjYjnI+LMmjaet70UEZdGxNKIuL+qbHREzIyIR8rHHXs49uSyzSMRcXLrou58PfTrP0fEQ+X/7zdGxA49HLvR747NXQ99e25EPF31//xRPRy70euJzV0PfXtNVb8+ERH39nCs522TOM1pgCJiC+Bh4DBgIXAXcEJmPlDV5u+AvTPzYxExHTguMz/QloAHkYjYGdg5M++JiO2Au4Fja/r2YODszHxvm8IctCLiCWBaZta9F3f5j90ngaOAA4BvZeYBrYtw8Cu/H54GDsjMJ6vKD8bztlci4h3ASuCKzPzLsuxCYHlmfrW84NoxMz9bc9xooAuYBiTF98d+mflMS3+BDtVDvx4O/EdmromI/wlQ269luyfYyHfH5q6Hvj0XWJmZX9vIcZu8ntjc1evbmvqvA89l5nl16p7A87YpHJkYuP2B+Zn5WGa+BFwNHFPT5hjg8vL59cChEREtjHFwlUHkAAAGvklEQVRQyszFmXlP+XwF8CAwsb1RbVaOofjCzsz8HbBDmeCp9w4FHq1OJNQ3mfkbYHlNcfV36uXAsXUOfTcwMzOXlwnETOCIpgU6yNTr18z8RWauKV/+DpjU8sCGgB7O2d7ozfXEZm1jfVteV/0tcFVLg5LJRANMBBZUvV7Ihhe8f2pTflE/B4xpSXRDRDk17I3AnXWq3xIRsyPi5oj4i5YGNrgl8IuIuDsiTq9T35tzWxs3nZ7/YfO87b8JmbkYij86AOPrtPH8HZhTgZt7qNvUd4fqO6OcQnZpD1PzPGcH5u3Aksx8pId6z9smMZkYuHojDLVzx3rTRj2IiG2B/wOcmZnP11TfQ7HF+z7A/wL+vdXxDWIHZua+wJHAJ8rh42qetwMQESOAo4Hr6lR73jaf528/RcQ/AGuAH/XQZFPfHdrQd4DXAG8AFgNfr9PGc3ZgTmDjoxKet01iMjFwC4Fdq15PAhb11CYitgReRf+GQDc7ETGcIpH4UWbeUFufmc9n5sry+QxgeESMbXGYg1JmLioflwI3UgyxV+vNua2eHQnck5lLais8bwdsyfopd+Xj0jptPH/7oVyo/l7gpOxhUWUvvjtUIzOXZObazFwH/Bv1+8xztp/Ka6v3Adf01MbztnlMJgbuLmDPiNi9/EvkdOCmmjY3AevvJPJ+igVu/rVhE8r5j98HHszMb/TQZqf1608iYn+Kc/oPrYtycIqIUeWidiJiFHA4cH9Ns5uA/xKFN1Msalvc4lAHsx7/SuZ5O2DV36knAz+p0+ZW4PCI2LGcUnJ4WaYeRMQRwGeBozNzVQ9tevPdoRo1682Oo36f9eZ6QvW9C3goMxfWq/S8ba4t2x3AYFfe9eIMin+ktgAuzcy5EXEe0JWZN1FcEF8ZEfMpRiSmty/iQeVA4EPAfVW3evsCMBkgM79LkZx9PCLWAC8A003UemUCcGN5Pbsl8OPMvCUiPgZ/6tsZFHdymg+sAj7cplgHnYjYhuKOLB+tKqvuW8/bXoqIq4CDgbERsRD4EvBV4NqIOA14Cji+bDsN+Fhm/tfMXB4RX6a4QAM4LzMdES710K+fB0YCM8vvht+VdyHcBfjfmXkUPXx3tOFX6Fg99O3BEfEGimlLT1B+N1T3bU/XE234FTpWvb7NzO9TZ32a523reGtYSZIkSf3iNCdJkiRJ/WIyIUmSJKlfTCYkSZIk9YvJhCRJkqR+MZmQJEmS1C8mE5KkjhYRu0VElrd+lSR1EJMJSZIkSf1iMiFJkiSpX0wmJEkbFYXPRMSjEfFCRNwXER8s69ZPQToxIm6PiNUR8VBEHF7zHu+IiDvL+iUR8c2IGFHzGWdFxCMR8WJELIyIC2pCmRIRMyNiVUQ8EBGHVR0/PCIuiohF5fELIuKrTe0YSZLJhCRpk/4JOA34BDAVuAD4XkS8p6rNhcBFwBuAmcBPImIiQPl4MzALeGP5XieU77Pe+cA/lmV/ARwPLKiJ4yvlZ+wD3AVcHRHblnWfAo4DpgN7Ah8A5g3w95YkbUJkZrtjkCR1qIgYBSwDDs/M31aV/wvwWuDvgMeBczLzK2XdMOAh4NrMPCcivkJxcf/azFxXtjkF+B6wI8UftpYBZ2bmd+vEsFv5GR/LzO+VZROBhcDbM/P2iLiIIgl5V/oPmyS1zJbtDkCS1NGmAlsBt0RE9UX6cOCJqtd3rH+Smesi4s7yWIDXA3esTyRKtwMjgD3K9x8J/HITscyper6ofBxfPl5GMSLycET8ApgB3FzzmZKkBjOZkCRtzPrpsH8NPFVT9zIQvXiPAHoaLchevsf6zysOysyI+FN8mXlPOYJxBHAIcDkwOyIOM6GQpOZxzYQkaWMeAF4EpmTm/JqfJ6vavXn9kyiu8vcHHqx6j7eU05/WexvwEvBo1WccOpBAM3NFZl6XmR8H3kORVOwxkPeUJG2cIxOSpB5l5oqI+BrwtTJJ+A2wLUXysA74Rdn04xHxMHAfxTqKKcB3yrqLgTOBiyPiW8Crga8C/5qZqwDK8gsi4sXyM8YA+2Xm+vfYqIj4NLAYuJdiBONE4HmKdRWSpCYxmZAkbco/AkuAsykShOcpLtovrGrzOeDTwL7Ak8BxmbkQIDOfjogjgX8uj3sW+DHwharjPw88U37WpPLzruhDjCuA/05xJ6ekuHPUkeuTFUlSc3g3J0lSv1XdaelNmdnV3mgkSa3mmglJkiRJ/WIyIUmSJKlfnOYkSZIkqV8cmZAkSZLULyYTkiRJkvrFZEKSJElSv5hMSJIkSeoXkwlJkiRJ/WIyIUmSJKlf/j+NCXU/jmnCTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 936x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "os.makedirs('./imgs', exist_ok=True)\n",
    "metrics = run.get_metrics()\n",
    "\n",
    "plt.figure(figsize = (13,5))\n",
    "plt.plot(metrics['validation_acc'], 'r-', lw=4, alpha=.6)\n",
    "plt.plot(metrics['training_acc'], 'b--', alpha=0.5)\n",
    "plt.legend(['Full evaluation set', 'Training set mini-batch'])\n",
    "plt.xlabel('epochs', fontsize=14)\n",
    "plt.ylabel('accuracy', fontsize=14)\n",
    "plt.title('Accuracy over Epochs', fontsize=16)\n",
    "run.log_image(name='acc_over_epochs.png', plot=plt)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the saved model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the training script, a TensorFlow `saver` object is used to persist the model in a local folder (local to the compute target). The model was saved to the `./outputs` folder on the disk of the Batch AI cluster node where the job is run. Azure ML automatically uploaded anything written in the `./outputs` folder into run history file store. Subsequently, we can use the `Run` object to download the model files the `saver` object saved. They are under the the `outputs/model` folder in the run history file store, and are downloaded into a local folder named `model`. Note the TensorFlow model consists of four files in binary format and they are not human-readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading from outputs/model/checkpoint to ./model/checkpoint ...\n",
      "Downloading from outputs/model/mnist-tf.model.data-00000-of-00001 to ./model/mnist-tf.model.data-00000-of-00001 ...\n",
      "Downloading from outputs/model/mnist-tf.model.index to ./model/mnist-tf.model.index ...\n",
      "Downloading from outputs/model/mnist-tf.model.meta to ./model/mnist-tf.model.meta ...\n"
     ]
    }
   ],
   "source": [
    "# create a model folder in the current directory\n",
    "os.makedirs('./model', exist_ok=True)\n",
    "\n",
    "for f in run.get_file_names():\n",
    "    if f.startswith('outputs/model'):\n",
    "        output_file_path = os.path.join('./model', f.split('/')[-1])\n",
    "        print('Downloading from {} to {} ...'.format(f, output_file_path))\n",
    "        run.download_file(name=f, output_file_path=output_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict on the test set\n",
    "Now load the saved TensorFlow graph, and list all operations under the `network` scope. This way we can discover the input tensor `network/X:0` and the output tensor `network/output/MatMul:0`, and use them in the scoring script in the next step.\n",
    "\n",
    "Note: if your local TensorFlow version is different than the version running in the cluster where the model is trained, you might see a \"compiletime version mismatch\" warning. You can ignore it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intelligent hyperparameter tuning\n",
    "We have trained the model with one set of hyperparameters, now let's how we can do hyperparameter tuning by launching multiple runs on the cluster. First let's define the parameter space using random sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.hyperdrive import RandomParameterSampling, BanditPolicy, HyperDriveConfig, PrimaryMetricGoal\n",
    "from azureml.train.hyperdrive import choice, loguniform\n",
    "\n",
    "ps = RandomParameterSampling(\n",
    "    {\n",
    "        '--batch-size': choice(25, 50, 100),\n",
    "        '--first-layer-neurons': choice(10, 50, 200, 300, 500),\n",
    "        '--second-layer-neurons': choice(10, 50, 200, 500),\n",
    "        '--learning-rate': loguniform(-6, -1)\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will create a new estimator without the above parameters since they will be passed in later. Note we still need to keep the `data-folder` parameter since that's not a hyperparamter we will sweep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - framework_version is not specified, defaulting to version 1.13.\n"
     ]
    }
   ],
   "source": [
    "est = TensorFlow(source_directory=script_folder,\n",
    "                 script_params={'--data-folder': dataset.as_named_input('mnist').as_mount()},\n",
    "                 compute_target=compute_target,\n",
    "                 entry_script='tf_mnist.py', \n",
    "                 use_gpu=True,\n",
    "                 pip_packages=['azureml-dataprep[pandas,fuse]'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will define an early termnination policy. The `BanditPolicy` basically states to check the job every 2 iterations. If the primary metric (defined later) falls outside of the top 10% range, Azure ML terminate the job. This saves us from continuing to explore hyperparameters that don't show promise of helping reach our target metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "policy = BanditPolicy(evaluation_interval=2, slack_factor=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to configure a run configuration object, and specify the primary metric `validation_acc` that's recorded in your training runs. If you go back to visit the training script, you will notice that this value is being logged after every epoch (a full batch set). We also want to tell the service that we are looking to maximizing this value. We also set the number of samples to 20, and maximal concurrent job to 4, which is the same as the number of nodes in our computer cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "htc = HyperDriveConfig(estimator=est, \n",
    "                       hyperparameter_sampling=ps, \n",
    "                       policy=policy, \n",
    "                       primary_metric_name='validation_acc', \n",
    "                       primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, \n",
    "                       max_total_runs=8,\n",
    "                       max_concurrent_runs=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's launch the hyperparameter tuning job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "htr = exp.submit(config=htc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use a run history widget to show the progress. Be patient as this might take a while to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ded9ad3c84ba4634a09270c2ac1719a4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_HyperDriveWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': True, 'log_level': 'INFO',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Running\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/tf-mnist/runs/tf-mnist_1573637336396137?wsid=/subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourcegroups/azuremlserviceresourcegroup/workspaces/azuremlservice\", \"run_id\": \"tf-mnist_1573637336396137\", \"run_properties\": {\"run_id\": \"tf-mnist_1573637336396137\", \"created_utc\": \"2019-11-13T09:28:56.53645Z\", \"properties\": {\"primary_metric_config\": \"{\\\"name\\\": \\\"validation_acc\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"resume_from\": \"null\", \"runTemplate\": \"HyperDrive\", \"azureml.runsource\": \"hyperdrive\", \"platform\": \"AML\", \"baggage\": \"eyJvaWQiOiAiZmY0MTM3YTgtMmQzNy00ZWU3LTg2NzUtOTg3NmViYTQ2OWNlIiwgInRpZCI6ICI3MmY5ODhiZi04NmYxLTQxYWYtOTFhYi0yZDdjZDAxMWRiNDciLCAidW5hbWUiOiAiMDRiMDc3OTUtOGRkYi00NjFhLWJiZWUtMDJmOWUxYmY3YjQ2In0\", \"ContentSnapshotId\": \"2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850\"}, \"tags\": {\"max_concurrent_jobs\": \"4\", \"max_total_jobs\": \"8\", \"max_duration_minutes\": \"10080\", \"policy_config\": \"{\\\"name\\\": \\\"BANDIT\\\", \\\"properties\\\": {\\\"evaluation_interval\\\": 2, \\\"delay_evaluation\\\": 0, \\\"slack_factor\\\": 0.1}}\", \"generator_config\": \"{\\\"name\\\": \\\"RANDOM\\\", \\\"parameter_space\\\": {\\\"--batch-size\\\": [\\\"choice\\\", [[25, 50, 100]]], \\\"--first-layer-neurons\\\": [\\\"choice\\\", [[10, 50, 200, 300, 500]]], \\\"--second-layer-neurons\\\": [\\\"choice\\\", [[10, 50, 200, 500]]], \\\"--learning-rate\\\": [\\\"loguniform\\\", [-6, -1]]}}\", \"primary_metric_config\": \"{\\\"name\\\": \\\"validation_acc\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"platform_config\": \"{\\\"ServiceAddress\\\": \\\"https://westeurope.experiments.azureml.net\\\", \\\"ServiceArmScope\\\": \\\"subscriptions/70b8f39e-8863-49f7-b6ba-34a80799550c/resourceGroups/azuremlserviceresourcegroup/providers/Microsoft.MachineLearningServices/workspaces/azuremlservice/experiments/tf-mnist\\\", \\\"SubscriptionId\\\": \\\"70b8f39e-8863-49f7-b6ba-34a80799550c\\\", \\\"ResourceGroupName\\\": \\\"azuremlserviceresourcegroup\\\", \\\"WorkspaceName\\\": \\\"azuremlservice\\\", \\\"ExperimentName\\\": \\\"tf-mnist\\\", \\\"Definition\\\": {\\\"Overrides\\\": {\\\"script\\\": \\\"tf_mnist.py\\\", \\\"arguments\\\": [\\\"--data-folder\\\", \\\"DatasetConsumptionConfig:mnist\\\"], \\\"target\\\": \\\"gpu-clustertf\\\", \\\"framework\\\": \\\"Python\\\", \\\"communicator\\\": \\\"None\\\", \\\"maxRunDurationSeconds\\\": null, \\\"nodeCount\\\": 1, \\\"environment\\\": {\\\"name\\\": null, \\\"version\\\": null, \\\"environmentVariables\\\": {\\\"EXAMPLE_ENV_VAR\\\": \\\"EXAMPLE_VALUE\\\"}, \\\"python\\\": {\\\"userManagedDependencies\\\": false, \\\"interpreterPath\\\": \\\"python\\\", \\\"condaDependenciesFile\\\": null, \\\"baseCondaEnvironment\\\": null, \\\"condaDependencies\\\": {\\\"name\\\": \\\"project_environment\\\", \\\"dependencies\\\": [\\\"python=3.6.2\\\", {\\\"pip\\\": [\\\"azureml-dataprep[pandas,fuse]\\\", \\\"azureml-defaults\\\", \\\"tensorflow-gpu==1.13.1\\\", \\\"horovod==0.16.1\\\"]}], \\\"channels\\\": [\\\"conda-forge\\\"]}}, \\\"docker\\\": {\\\"enabled\\\": true, \\\"baseImage\\\": \\\"mcr.microsoft.com/azureml/base-gpu:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04\\\", \\\"baseDockerfile\\\": null, \\\"sharedVolumes\\\": true, \\\"shmSize\\\": \\\"2g\\\", \\\"arguments\\\": [], \\\"baseImageRegistry\\\": {\\\"address\\\": null, \\\"username\\\": null, \\\"password\\\": null}}, \\\"spark\\\": {\\\"repositories\\\": [], \\\"packages\\\": [], \\\"precachePackages\\\": false}, \\\"databricks\\\": {\\\"mavenLibraries\\\": [], \\\"pypiLibraries\\\": [], \\\"rcranLibraries\\\": [], \\\"jarLibraries\\\": [], \\\"eggLibraries\\\": []}, \\\"inferencingStackVersion\\\": null}, \\\"history\\\": {\\\"outputCollection\\\": true, \\\"snapshotProject\\\": true, \\\"directoriesToWatch\\\": [\\\"logs\\\"]}, \\\"spark\\\": {\\\"configuration\\\": {\\\"spark.app.name\\\": \\\"Azure ML Experiment\\\", \\\"spark.yarn.maxAppAttempts\\\": 1}}, \\\"hdi\\\": {\\\"yarnDeployMode\\\": \\\"cluster\\\"}, \\\"tensorflow\\\": {\\\"workerCount\\\": 1, \\\"parameterServerCount\\\": 1}, \\\"mpi\\\": {\\\"processCountPerNode\\\": 1}, \\\"dataReferences\\\": {}, \\\"data\\\": {\\\"mnist\\\": {\\\"dataLocation\\\": {\\\"dataset\\\": {\\\"id\\\": \\\"3afaa0f2-1b8d-4b73-bf27-03d95e1d24f4\\\"}, \\\"datapath\\\": null}, \\\"createOutputDirectories\\\": false, \\\"mechanism\\\": \\\"mount\\\", \\\"environmentVariableName\\\": \\\"mnist\\\", \\\"pathOnCompute\\\": null, \\\"overwrite\\\": false}}, \\\"sourceDirectoryDataStore\\\": null, \\\"amlcompute\\\": {\\\"vmSize\\\": null, \\\"vmPriority\\\": null, \\\"retainCluster\\\": false, \\\"name\\\": null, \\\"clusterMaxNodeCount\\\": 1}}, \\\"TargetDetails\\\": null, \\\"SnapshotId\\\": \\\"2c1bd6ab-e3b2-49ac-8fc7-06a36c3bc850\\\", \\\"TelemetryValues\\\": {\\\"amlClientType\\\": \\\"azureml-sdk-train\\\", \\\"amlClientModule\\\": \\\"azureml.train.hyperdrive._search\\\", \\\"amlClientFunction\\\": \\\"search\\\", \\\"tenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\", \\\"amlClientRequestId\\\": \\\"3238bb7a-8fd7-4e11-9282-d07ca97036aa\\\", \\\"amlClientSessionId\\\": \\\"b85ff7f2-6d32-4347-84bf-e5c69c4a1391\\\", \\\"subscriptionId\\\": \\\"70b8f39e-8863-49f7-b6ba-34a80799550c\\\", \\\"estimator\\\": \\\"TensorFlow\\\", \\\"samplingMethod\\\": \\\"RANDOM\\\", \\\"terminationPolicy\\\": \\\"Bandit\\\", \\\"primaryMetricGoal\\\": \\\"maximize\\\", \\\"maxTotalRuns\\\": 8, \\\"maxConcurrentRuns\\\": 4, \\\"maxDurationMinutes\\\": 10080, \\\"computeTarget\\\": \\\"AmlCompute\\\", \\\"vmSize\\\": null}}}\", \"resume_child_runs\": \"null\", \"all_jobs_generated\": \"true\", \"cancellation_requested\": \"false\", \"progress_metadata_evaluation_timestamp\": \"\\\"2019-11-13T09:28:57.155827\\\"\", \"progress_metadata_digest\": \"\\\"29cc97915801bebd31d6c8910f9c7f51e0c0a58560a59ede94a30efae722f208\\\"\", \"progress_metadata_active_timestamp\": \"\\\"2019-11-13T09:28:57.155827\\\"\", \"environment_preparation_status\": \"PREPARED\", \"prepare_run_id\": \"tf-mnist_1573637336396137_preparation\", \"tf-mnist_1573637336396137_0\": \"{\\\"--batch-size\\\": 50, \\\"--first-layer-neurons\\\": 10, \\\"--learning-rate\\\": 0.006206542987939142, \\\"--second-layer-neurons\\\": 200}\", \"tf-mnist_1573637336396137_1\": \"{\\\"--batch-size\\\": 100, \\\"--first-layer-neurons\\\": 50, \\\"--learning-rate\\\": 0.2609500250793897, \\\"--second-layer-neurons\\\": 10}\", \"tf-mnist_1573637336396137_2\": \"{\\\"--batch-size\\\": 25, \\\"--first-layer-neurons\\\": 50, \\\"--learning-rate\\\": 0.03830593051850398, \\\"--second-layer-neurons\\\": 50}\", \"tf-mnist_1573637336396137_3\": \"{\\\"--batch-size\\\": 50, \\\"--first-layer-neurons\\\": 500, \\\"--learning-rate\\\": 0.00466124470375142, \\\"--second-layer-neurons\\\": 500}\", \"tf-mnist_1573637336396137_4\": \"{\\\"--batch-size\\\": 100, \\\"--first-layer-neurons\\\": 10, \\\"--learning-rate\\\": 0.002524054356458926, \\\"--second-layer-neurons\\\": 500}\", \"tf-mnist_1573637336396137_5\": \"{\\\"--batch-size\\\": 50, \\\"--first-layer-neurons\\\": 500, \\\"--learning-rate\\\": 0.014960636623360057, \\\"--second-layer-neurons\\\": 200}\", \"tf-mnist_1573637336396137_6\": \"{\\\"--batch-size\\\": 25, \\\"--first-layer-neurons\\\": 300, \\\"--learning-rate\\\": 0.017515078581028635, \\\"--second-layer-neurons\\\": 500}\", \"tf-mnist_1573637336396137_7\": \"{\\\"--batch-size\\\": 50, \\\"--first-layer-neurons\\\": 300, \\\"--learning-rate\\\": 0.3577290620459196, \\\"--second-layer-neurons\\\": 200}\"}, \"end_time_utc\": null, \"status\": \"Running\", \"log_files\": {\"azureml-logs/hyperdrive.txt\": \"https://azuremlservice8628362969.blob.core.windows.net/azureml/ExperimentRun/dcid.tf-mnist_1573637336396137/azureml-logs/hyperdrive.txt?sv=2019-02-02&sr=b&sig=EnA7ezyLJ%2B94acZzePkqGKRap2zVjaLKE3HUHefjYoA%3D&st=2019-11-13T09%3A27%3A29Z&se=2019-11-13T17%3A37%3A29Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/hyperdrive.txt\"]], \"run_duration\": \"0:08:32\", \"hyper_parameters\": {\"--batch-size\": [\"choice\", [[25, 50, 100]]], \"--first-layer-neurons\": [\"choice\", [[10, 50, 200, 300, 500]]], \"--second-layer-neurons\": [\"choice\", [[10, 50, 200, 500]]], \"--learning-rate\": [\"loguniform\", [-6, -1]]}}, \"child_runs\": [{\"run_id\": \"tf-mnist_1573637336396137_2\", \"run_number\": 6, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:31:22.3036Z\", \"end_time\": \"2019-11-13T09:33:24.228602Z\", \"created_time\": \"2019-11-13T09:29:28.524405Z\", \"created_time_dt\": \"2019-11-13T09:29:28.524405Z\", \"duration\": \"0:03:55\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 25, \"param_--first-layer-neurons\": 50, \"param_--learning-rate\": 0.03830593051850398, \"param_--second-layer-neurons\": 50, \"best_metric\": 0.9772999882698059}, {\"run_id\": \"tf-mnist_1573637336396137_1\", \"run_number\": 7, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:33:25.453317Z\", \"end_time\": \"2019-11-13T09:34:47.117472Z\", \"created_time\": \"2019-11-13T09:29:28.574649Z\", \"created_time_dt\": \"2019-11-13T09:29:28.574649Z\", \"duration\": \"0:05:18\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 100, \"param_--first-layer-neurons\": 50, \"param_--learning-rate\": 0.2609500250793897, \"param_--second-layer-neurons\": 10, \"best_metric\": 0.9733999967575073}, {\"run_id\": \"tf-mnist_1573637336396137_3\", \"run_number\": 8, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:29:39.808824Z\", \"end_time\": \"2019-11-13T09:31:20.420894Z\", \"created_time\": \"2019-11-13T09:29:28.570981Z\", \"created_time_dt\": \"2019-11-13T09:29:28.570981Z\", \"duration\": \"0:01:51\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 50, \"param_--first-layer-neurons\": 500, \"param_--learning-rate\": 0.00466124470375142, \"param_--second-layer-neurons\": 500, \"best_metric\": 0.9602000117301941}, {\"run_id\": \"tf-mnist_1573637336396137_0\", \"run_number\": 9, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:34:48.698098Z\", \"end_time\": \"2019-11-13T09:36:20.476788Z\", \"created_time\": \"2019-11-13T09:29:29.43846Z\", \"created_time_dt\": \"2019-11-13T09:29:29.43846Z\", \"duration\": \"0:06:51\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 50, \"param_--first-layer-neurons\": 10, \"param_--learning-rate\": 0.006206542987939142, \"param_--second-layer-neurons\": 200, \"best_metric\": 0.9320999979972839}, {\"run_id\": \"tf-mnist_1573637336396137_4\", \"run_number\": 10, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:34:54.86028Z\", \"end_time\": \"\", \"created_time\": \"2019-11-13T09:31:34.386479Z\", \"created_time_dt\": \"2019-11-13T09:31:34.386479Z\", \"duration\": \"0:05:55\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 100, \"param_--first-layer-neurons\": 10, \"param_--learning-rate\": 0.002524054356458926, \"param_--second-layer-neurons\": 500}, {\"run_id\": \"tf-mnist_1573637336396137_5\", \"run_number\": 11, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:36:02.22989Z\", \"end_time\": \"\", \"created_time\": \"2019-11-13T09:33:36.761647Z\", \"created_time_dt\": \"2019-11-13T09:33:36.761647Z\", \"duration\": \"0:03:52\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 50, \"param_--first-layer-neurons\": 500, \"param_--learning-rate\": 0.014960636623360057, \"param_--second-layer-neurons\": 200}, {\"run_id\": \"tf-mnist_1573637336396137_6\", \"run_number\": 12, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:36:16.874078Z\", \"end_time\": \"\", \"created_time\": \"2019-11-13T09:35:09.009277Z\", \"created_time_dt\": \"2019-11-13T09:35:09.009277Z\", \"duration\": \"0:02:20\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 25, \"param_--first-layer-neurons\": 300, \"param_--learning-rate\": 0.017515078581028635, \"param_--second-layer-neurons\": 500}, {\"run_id\": \"tf-mnist_1573637336396137_7\", \"run_number\": 13, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2019-11-13T09:36:51.616913Z\", \"end_time\": \"\", \"created_time\": \"2019-11-13T09:36:40.831869Z\", \"created_time_dt\": \"2019-11-13T09:36:40.831869Z\", \"duration\": \"0:00:48\", \"hyperdrive_id\": \"1573637336396137\", \"arguments\": null, \"param_--batch-size\": 50, \"param_--first-layer-neurons\": 300, \"param_--learning-rate\": 0.3577290620459196, \"param_--second-layer-neurons\": 200}], \"children_metrics\": {\"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], \"series\": {\"training_acc\": [{\"run_id\": 8, \"name\": 8, \"data\": [0.8999999761581421, 0.9200000166893005, 0.9599999785423279, 0.8999999761581421, 0.8999999761581421, 0.9399999976158142, 0.9599999785423279, 0.9800000190734863, 0.9599999785423279, 0.9200000166893005, 0.9399999976158142, 0.8999999761581421, 0.9800000190734863, 0.9599999785423279, 0.9599999785423279, 0.9800000190734863, 0.9800000190734863, 0.9800000190734863, 0.9200000166893005, 1], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 6, \"name\": 6, \"data\": [0.9599999785423279, 1, 1, 1, 1, 1, 0.9599999785423279, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 7, \"name\": 7, \"data\": [0.9599999785423279, 0.9900000095367432, 0.9700000286102295, 0.9900000095367432, 1, 0.9900000095367432, 1, 1, 0.9900000095367432, 0.9900000095367432, 1, 1, 1, 0.9900000095367432, 1, 0.9900000095367432, 1, 1, 1, 1], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 9, \"name\": 9, \"data\": [0.7400000095367432, 0.8799999952316284, 0.9200000166893005, 0.8399999737739563, 0.800000011920929, 0.9399999976158142, 0.9399999976158142, 0.9800000190734863, 0.8799999952316284, 0.9599999785423279, 1, 0.9200000166893005, 0.9599999785423279, 0.9800000190734863, 0.9800000190734863, 0.9800000190734863, 1, 0.9200000166893005, 0.8999999761581421, 0.9599999785423279], \"mode\": \"lines\", \"stepped\": false}], \"validation_acc\": [{\"run_id\": 8, \"name\": 8, \"data\": [0.882099986076355, 0.9064000248908997, 0.9169999957084656, 0.9253000020980835, 0.9284999966621399, 0.9337000250816345, 0.9358000159263611, 0.9376000165939331, 0.941100001335144, 0.9437000155448914, 0.9469000101089478, 0.9473000168800354, 0.9496999979019165, 0.9503999948501587, 0.9531999826431274, 0.9549000263214111, 0.9567999839782715, 0.9588000178337097, 0.9578999876976013, 0.9602000117301941], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 6, \"name\": 6, \"data\": [0.9429000020027161, 0.9527000188827515, 0.9610999822616577, 0.9659000039100647, 0.9674000144004822, 0.9684000015258789, 0.9707000255584717, 0.9707000255584717, 0.9740999937057495, 0.9743000268936157, 0.9742000102996826, 0.9745000004768372, 0.9742000102996826, 0.975600004196167, 0.974399983882904, 0.9746999740600586, 0.9757000207901001, 0.9764999747276306, 0.9772999882698059, 0.9743000268936157], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 7, \"name\": 7, \"data\": [0.9412000179290771, 0.9519000053405762, 0.95660001039505, 0.961899995803833, 0.9663000106811523, 0.9700999855995178, 0.9679999947547913, 0.9681000113487244, 0.97079998254776, 0.9661999940872192, 0.9718000292778015, 0.9711999893188477, 0.972599983215332, 0.9696999788284302, 0.9713000059127808, 0.9728999733924866, 0.9677000045776367, 0.9733999967575073, 0.967199981212616, 0.972000002861023], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 9, \"name\": 9, \"data\": [0.7660999894142151, 0.8693000078201294, 0.8913999795913696, 0.901199996471405, 0.9093000292778015, 0.9115999937057495, 0.9161999821662903, 0.9186999797821045, 0.9208999872207642, 0.9222999811172485, 0.9240000247955322, 0.9262999892234802, 0.9264000058174133, 0.9286999702453613, 0.9284999966621399, 0.9294000267982483, 0.9298999905586243, 0.9320999979972839, 0.9307000041007996, 0.9320999979972839], \"mode\": \"lines\", \"stepped\": false}], \"final_acc\": [{\"run_id\": 8, \"name\": 8, \"data\": [0.9602000117301941], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 6, \"name\": 6, \"data\": [0.9743000268936157], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 7, \"name\": 7, \"data\": [0.972000002861023], \"mode\": \"lines\", \"stepped\": false}, {\"run_id\": 9, \"name\": 9, \"data\": [0.9320999979972839], \"mode\": \"lines\", \"stepped\": false}]}, \"metricName\": null, \"primaryMetricName\": \"validation_acc\", \"showLegend\": true}, \"run_metrics\": [{\"name\": \"best_child_by_primary_metric\", \"run_id\": \"tf-mnist_1573637336396137\", \"categories\": [0], \"series\": [{\"data\": [{\"metric_name\": [\"validation_acc\", \"validation_acc\", \"validation_acc\", \"validation_acc\"], \"timestamp\": [\"2019-11-13 09:30:57.108022+00:00\", \"2019-11-13 09:31:29.156596+00:00\", \"2019-11-13 09:33:04.104756+00:00\", \"2019-11-13 09:33:35.594398+00:00\"], \"run_id\": [\"tf-mnist_1573637336396137_3\", \"tf-mnist_1573637336396137_3\", \"tf-mnist_1573637336396137_2\", \"tf-mnist_1573637336396137_2\"], \"metric_value\": [0.9358000159263611, 0.9602000117301941, 0.9740999937057495, 0.9772999882698059], \"final\": [false, false, false, false]}]}]}], \"run_logs\": \"[2019-11-13T09:28:56.757607][API][INFO]Experiment created\\r\\n[2019-11-13T09:28:57.4408299Z][SCHEDULER][INFO]The execution environment is being prepared. Please be patient as it can take a few minutes.\\r\\n[2019-11-13T09:28:57.538000][GENERATOR][INFO]Trying to sample '4' jobs from the hyperparameter space\\r\\n[2019-11-13T09:28:57.663200][GENERATOR][INFO]Successfully sampled '4' jobs, they will soon be submitted to the execution target.\\r\\n[2019-11-13T09:29:27.8925843Z][SCHEDULER][INFO]The execution environment was successfully prepared.\\r\\n[2019-11-13T09:29:27.8980203Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_3'\\r\\n[2019-11-13T09:29:27.8965679Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_2'\\r\\n[2019-11-13T09:29:27.8949238Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_1'\\r\\n[2019-11-13T09:29:27.8932289Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_0'\\r\\n[2019-11-13T09:29:28.6150321Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_2'\\r\\n[2019-11-13T09:29:28.6616682Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_3'\\r\\n[2019-11-13T09:29:28.6357641Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_1'\\r\\n[2019-11-13T09:29:29.5074077Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_0'\\r\\n[2019-11-13T09:29:56.782616][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_3', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:30:26.879644][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_3', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:31:27.012429][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_2', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:31:27.165212][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2019-11-13T09:31:27.021463][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2019-11-13T09:31:31.7959612Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_4'\\r\\n[2019-11-13T09:31:34.6096340Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_4'\\r\\n[2019-11-13T09:31:57.009063][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_2', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:32:27.107791][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_2', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:33:27.011868][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_1', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:33:27.146177][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2019-11-13T09:33:27.269630][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2019-11-13T09:33:35.6063652Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_5'\\r\\n[2019-11-13T09:33:36.8538194Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_5'\\r\\n[2019-11-13T09:33:56.921446][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_1', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:34:58.428854][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2019-11-13T09:34:58.224053][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2019-11-13T09:34:59.010913][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_0', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_4', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:35:08.1509303Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_6'\\r\\n[2019-11-13T09:35:09.1103623Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_6'\\r\\n[2019-11-13T09:35:28.823510][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_0', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_4', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:35:58.985592][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_4', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:36:28.656255][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_4', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_5', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_6', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:36:29.421510][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2019-11-13T09:36:29.533620][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2019-11-13T09:36:40.0321768Z][SCHEDULER][INFO]Scheduling job, id='tf-mnist_1573637336396137_7'\\r\\n[2019-11-13T09:36:40.9231351Z][SCHEDULER][INFO]Successfully scheduled a job. Id='tf-mnist_1573637336396137_7'\\r\\n[2019-11-13T09:36:58.567918][ENFORCER][INFO]Jobs [RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_4', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_5', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_6', data_container_id=None), RunKey(run_scope=RunScope(host='https://westeurope.experiments.azureml.net', subscription_id='70b8f39e-8863-49f7-b6ba-34a80799550c', resource_group='azuremlserviceresourcegroup', workspace_name='azuremlservice', project_name='tf-mnist'), run_id='tf-mnist_1573637336396137_7', data_container_id=None)] do not contain any metrics named 'validation_acc' at this moment, policy cannot be applied.\\r\\n[2019-11-13T09:36:59.913181][GENERATOR][INFO]Max number of jobs '8' reached for experiment.\\r\\n[2019-11-13T09:37:00.237009][GENERATOR][INFO]All jobs generated.\\n\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": true, \"log_level\": \"INFO\", \"sdk_version\": \"1.0.72\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "RunDetails(htr).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#htr.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warm start a Hyperparameter Tuning experiment and resuming child runs\n",
    "Often times, finding the best hyperparameter values for your model can be an iterative process, needing multiple tuning runs that learn from previous hyperparameter tuning runs. Reusing knowledge from these previous runs will accelerate the hyperparameter tuning process, thereby reducing the cost of tuning the model and will potentially improve the primary metric of the resulting model. When warm starting a hyperparameter tuning experiment with Bayesian sampling, trials from the previous run will be used as prior knowledge to intelligently pick new samples, so as to improve the primary metric. Additionally, when using Random or Grid sampling, any early termination decisions will leverage metrics from the previous runs to determine poorly performing training runs. \n",
    "\n",
    "Azure Machine Learning allows you to warm start your hyperparameter tuning run by leveraging knowledge from up to 5 previously completed hyperparameter tuning parent runs. \n",
    "\n",
    "Additionally, there might be occasions when individual training runs of a hyperparameter tuning experiment are cancelled due to budget constraints or fail due to other reasons. It is now possible to resume such individual training runs from the last checkpoint (assuming your training script handles checkpoints). Resuming an individual training run will use the same hyperparameter configuration and mount the storage used for that run. The training script should accept the \"--resume-from\" argument, which contains the checkpoint or model files from which to resume the training run. You can also resume individual runs as part of an experiment that spends additional budget on hyperparameter tuning. Any additional budget, after resuming the specified training runs is used for exploring additional configurations.\n",
    "\n",
    "For more information on warm starting and resuming hyperparameter tuning runs, please refer to the [Hyperparameter Tuning for Azure Machine Learning documentation](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters) \n",
    "\n",
    "## Find and register best model <a class=\"anchor\" id=\"register-model\"></a>\n",
    "When all the jobs finish, we can find out the one that has the highest accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run = htr.get_best_run_by_primary_metric()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's list the model files uploaded during the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(best_run.get_file_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then register the folder (and all files in it) as a model named `tf-dnn-mnist` under the workspace for deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = best_run.register_model(model_name='tf-dnn-mnist', model_path='outputs/model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the model in ACI\n",
    "Now we are ready to deploy the model as a web service running in Azure Container Instance [ACI](https://azure.microsoft.com/en-us/services/container-instances/). Azure Machine Learning accomplishes this by constructing a Docker image with the scoring logic and model baked in.\n",
    "### Create score.py\n",
    "First, we will create a scoring script that will be invoked by the web service call. \n",
    "\n",
    "* Note that the scoring script must have two required functions, `init()` and `run(input_data)`. \n",
    "  * In `init()` function, you typically load the model into a global object. This function is executed only once when the Docker container is started. \n",
    "  * In `run(input_data)` function, the model is used to predict a value based on the input data. The input and output to `run` typically use JSON as serialization and de-serialization format but you are not limited to that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile score.py\n",
    "import json\n",
    "import numpy as np\n",
    "import os\n",
    "import tensorflow as tf\n",
    "\n",
    "from azureml.core.model import Model\n",
    "\n",
    "def init():\n",
    "    global X, output, sess\n",
    "    tf.reset_default_graph()\n",
    "    model_root = Model.get_model_path('tf-dnn-mnist')\n",
    "    saver = tf.train.import_meta_graph(os.path.join(model_root, 'mnist-tf.model.meta'))\n",
    "    X = tf.get_default_graph().get_tensor_by_name(\"network/X:0\")\n",
    "    output = tf.get_default_graph().get_tensor_by_name(\"network/output/MatMul:0\")\n",
    "    \n",
    "    sess = tf.Session()\n",
    "    saver.restore(sess, os.path.join(model_root, 'mnist-tf.model'))\n",
    "\n",
    "def run(raw_data):\n",
    "    data = np.array(json.loads(raw_data)['data'])\n",
    "    # make prediction\n",
    "    out = output.eval(session=sess, feed_dict={X: data})\n",
    "    y_hat = np.argmax(out, axis=1)\n",
    "    return y_hat.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create myenv.yml\n",
    "We also need to create an environment file so that Azure Machine Learning can install the necessary packages in the Docker image which are required by your scoring script. In this case, we need to specify packages `numpy`, `tensorflow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.runconfig import CondaDependencies\n",
    "\n",
    "cd = CondaDependencies.create()\n",
    "cd.add_conda_package('numpy')\n",
    "cd.add_tensorflow_conda_package()\n",
    "cd.save_to_file(base_directory='./', conda_file_path='myenv.yml')\n",
    "\n",
    "print(cd.serialize_to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy to ACI\n",
    "We are almost ready to deploy. Create the inference configuration and deployment configuration and deploy to ACI. This cell will run for about 7-8 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AciWebservice\n",
    "from azureml.core.model import InferenceConfig\n",
    "from azureml.core.webservice import Webservice\n",
    "from azureml.core.model import Model\n",
    "\n",
    "inference_config = InferenceConfig(runtime= \"python\", \n",
    "                                   entry_script=\"score.py\",\n",
    "                                   conda_file=\"myenv.yml\")\n",
    "\n",
    "aciconfig = AciWebservice.deploy_configuration(cpu_cores=1, \n",
    "                                               memory_gb=1, \n",
    "                                               tags={'name':'mnist', 'framework': 'TensorFlow DNN'},\n",
    "                                               description='Tensorflow DNN on MNIST')\n",
    "\n",
    "service = Model.deploy(workspace=ws, \n",
    "                           name='tf-mnist-svc', \n",
    "                           models=[model], \n",
    "                           inference_config=inference_config, \n",
    "                           deployment_config=aciconfig)\n",
    "\n",
    "service.wait_for_deployment(True)\n",
    "print(service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip: If something goes wrong with the deployment, the first thing to look at is the logs from the service by running the following command:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(service.get_logs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the scoring web service endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(service.scoring_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the deployed model\n",
    "Let's test the deployed model. Pick 30 random samples from the test set, and send it to the web service hosted in ACI. Note here we are using the `run` API in the SDK to invoke the service. You can also make raw HTTP calls using any HTTP tool such as curl.\n",
    "\n",
    "After the invocation, we print the returned predictions and plot them along with the input images. Use red font color and inversed image (white on black) to highlight the misclassified samples. Note since the model accuracy is pretty high, you might have to run the below cell a few times before you can see a misclassified sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# find 30 random samples from test set\n",
    "n = 30\n",
    "sample_indices = np.random.permutation(X_test.shape[0])[0:n]\n",
    "\n",
    "test_samples = json.dumps({\"data\": X_test[sample_indices].tolist()})\n",
    "test_samples = bytes(test_samples, encoding='utf8')\n",
    "\n",
    "# predict using the deployed model\n",
    "result = service.run(input_data=test_samples)\n",
    "\n",
    "# compare actual value vs. the predicted values:\n",
    "i = 0\n",
    "plt.figure(figsize = (20, 1))\n",
    "\n",
    "for s in sample_indices:\n",
    "    plt.subplot(1, n, i + 1)\n",
    "    plt.axhline('')\n",
    "    plt.axvline('')\n",
    "    \n",
    "    # use different color for misclassified sample\n",
    "    font_color = 'red' if y_test[s] != result[i] else 'black'\n",
    "    clr_map = plt.cm.gray if y_test[s] != result[i] else plt.cm.Greys\n",
    "    \n",
    "    plt.text(x=10, y=-10, s=y_hat[s], fontsize=18, color=font_color)\n",
    "    plt.imshow(X_test[s].reshape(28, 28), cmap=clr_map)\n",
    "    \n",
    "    i = i + 1\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also send raw HTTP request to the service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# send a random row from the test set to score\n",
    "random_index = np.random.randint(0, len(X_test)-1)\n",
    "input_data = \"{\\\"data\\\": [\" + str(list(X_test[random_index])) + \"]}\"\n",
    "\n",
    "headers = {'Content-Type':'application/json'}\n",
    "\n",
    "resp = requests.post(service.scoring_uri, input_data, headers=headers)\n",
    "\n",
    "print(\"POST to url\", service.scoring_uri)\n",
    "#print(\"input data:\", input_data)\n",
    "print(\"label:\", y_test[random_index])\n",
    "print(\"prediction:\", resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the workspace after the web service was deployed. You should see \n",
    "* a registered model named 'model' and with the id 'model:1'\n",
    "* a webservice called 'tf-mnist' with some scoring URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = ws.models\n",
    "for name, model in models.items():\n",
    "    print(\"Model: {}, ID: {}\".format(name, model.id))\n",
    "    \n",
    "webservices = ws.webservices\n",
    "for name, webservice in webservices.items():\n",
    "    print(\"Webservice: {}, scoring URI: {}\".format(name, webservice.scoring_uri))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "You can delete the ACI deployment with a simple delete API call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service.delete()"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "ninhu"
   }
  ],
  "category": "training",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "MNIST"
  ],
  "deployment": [
   "Azure Container Instance"
  ],
  "exclude_from_index": false,
  "framework": [
   "TensorFlow"
  ],
  "friendly_name": "Training and hyperparameter tuning using the TensorFlow estimator",
  "index_order": 1,
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "tags": [
   "None"
  ],
  "task": "Train a deep neural network"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}