{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear Classification\n",
"\n",
"- Algorithms that learn linear decision boundaries for classification tasks.\n",
"- Note that the model can be non-linear such as logistic regression or SVM. But the decision boundary is linear.\n",
"- The goal is to learn a hyper-plane $\\mathbf{x}^T \\mathbf{w} + b = 0$ to separate the date.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Least square classification\n",
"\n",
"In assignment 1, we used linear regression for classification:\n",
"$$y(\\mathbf{x}, \\mathbf{w}) = \\mathbf{x}^T \\mathbf{w} + b$$\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Logistic Regression Model \n",
"\n",
"We will consider linear model for classification. Note that the model is linear in parameters.\n",
"\n",
"$$y(\\mathbf{x}, \\mathbf{w}) = \\sigma (\\mathbf{x}^T \\mathbf{w} + b)$$\n",
"\n",
"where\n",
"\n",
"$$ \\sigma(x) = {1 \\over {1 + e^{-x}}}$$\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Logistic Regression Example"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Populating the interactive namespace from numpy and matplotlib\n"
]
}
],
"source": [
"import tensorflow as tf\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"%pylab inline\n",
"\n",
"import warnings\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Loading the dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"with np.load(\"TINY_MNIST.npz\") as data:\n",
" x_train, t_train = data[\"x\"], data[\"t\"]\n",
" x_eval, t_eval = data[\"x_eval\"], data[\"t_eval\"]"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABHMAAAJMCAYAAAB5Keo1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmgjfXC/v+PFDILGSIVmTMrpQmZSihDFEmUKE4aFKHS\nQEWGkiQ5hjJVopBo4EGSOfNRyRCOKbM47O8fz+853+85v+ta272tbVvneb/+O9dy3/dnrXVP69M+\n95UuKSkpKQAAAAAAACAhXJDWAwAAAAAAAMCZYzIHAAAAAAAggTCZAwAAAAAAkECYzAEAAAAAAEgg\nTOYAAAAAAAAkECZzAAAAAAAAEshZTeZ8+eWXoWTJkuHqq68Or732WrzGBAAAAAAAACNdUlJSUkoW\nPHXqVChRokSYM2dOuOyyy0LVqlXD+PHjQ6lSpeI9RgAAAAAAAPx/LkzpgosXLw7FihULV1xxRQgh\nhBYtWoSpU6f+y2ROunTpznqAAAAAAAAA/1upv8FJ8f/Navv27aFw4cL//N+FChUK27dvT+nqAAAA\nAAAAcAZSPJnDX90AAAAAAACceymezLnsssvC1q1b//m/t27dGgoVKhSXQQEAAAAAAEBL8WROlSpV\nwt/+9rewefPmcOLEiTBx4sTQsGHDeI4NAAAAAAAA/ybFD0C+8MILw9tvvx3q1q0bTp06Fdq1a5cm\nTVYXXKDnozp06CDzGjVqyDxbtmwy/+ijj2Q+duzYMxjdmcmXL5/Mb7/9dpmvXLlS5tdff73Mb775\nZpnnzZtX5jVr1pT5uZAhQwaZ33333TJv1KiRXdfOnTtl3rVr1+gDi6B69eoyf+SRR+wy//Mg8X9X\nvHhxmQ8YMEDmr7/+euzBnaGLL77Yvla5cmWZt27dWubu2HLfdcaMGWV+55132jEpRYsWta9t2rRJ\n5l9//bXMd+/eLfN//OMfkca0bNkymQ8cODDSemKpU6eOzGfMmCHzn376Sebbtm2Teffu3e22V69e\nnczo/lW5cuVk/swzz8g8d+7cMi9QoIDMf/vtN7vtkSNHynzq1Kl2mbTgzg0h/Pd/VFHc/l2rVi2Z\nV6tWTebNmjWLPbgI2rZtK/MWLVrI/NChQzKvUKGCzN05a9asWTJ/8MEHZX4uuPuQLl26yLx06dJ2\nXe74bdq0qczd+btbt24yz5Ejh8wvuugimadPn17mPXv2lHkIIcyfP9++dr5x7/udd96ReaVKlWTu\n9j93f+e4zzuEEFq1aiXzJ554QuarVq2S+SWXXCLzO+64I5nRnXtuP3b3tXv37pX5N998I/MxY8bY\nbZ88eTKZ0f0rd2/UqVMnmbvztJM/f377mrs/dvvx7NmzI207njJnzixzd1686667ZJ4nTx6Zu9+L\nacmdp929ortvcb8VzoWsWbPK/IEHHpD5tddeK/OcOXPabaTVH5G4+xn3WzUEf65esmSJzHv37n3G\n40nxZE4IIdSvXz/Ur1//bFYBAAAAAACACFL8f7MCAAAAAADAucdkDgAAAAAAQAJhMgcAAAAAACCB\nMJkDAAAAAACQQNIlJSUlpdrK06VLrVX/k2uzatKkiczdU/mfe+45mbsn3VesWPEMRvd/xWokWbRo\nkczd0/RdC5Br3FmwYIHMFy9eLPONGzfKPBY3pkGDBsn8zTffjDQm1zwzefJkOya3rtTmvuvNmzfH\nbRvusI3XMeda3EIIYenSpTJ37TMud8fWihUrZP73v/9d5uXLl5e5a7EJIYTHH39c5q5tKiXHRGor\nUqSIzN1+75rf3Hs+ceJEygZ2HnGtQSH486trHDh9+nRcxuS4FrdYLSI33XSTzH/99VeZr1u3Tuaj\nRo2S+ccff2y3rbjrawghbN++XeauueXgwYMyd21Crn3Ntba4xq94co16Q4cOlblrOIrVWDRp0iSZ\nu3ZLd0xs3bpV5lE/p7p168r80UcftcukVSOJE+t+zd1PZc+eXeauxcu1XjZv3jz24P6NayUNwV8j\nXHuK484nV155ZaT1nAuu8dA1R7nfCv3795e5a5kNIYSZM2cmM7p/5c75b7zxRqT1TJkyReYHDhyw\ny7hzkPsdkdrHaN++fe1r7vfWuHHjZP7dd9/J3F0j0pJrnHP7n/st5PaBqMd6PH344Ycydw3OX375\npcxjtUfv2rUr+sAicOeNLVu2yLxYsWJ2Xe5+wF075syZI3P1+4+/zAEAAAAAAEggTOYAAAAAAAAk\nECZzAAAAAAAAEgiTOQAAAAAAAAmEyRwAAAAAAIAEcmFaD+BMuLaaEEJ45JFHZF68eHGZX3XVVTJ3\nDT3PPvtsMqP7V1myZJH5ypUr7TLTpk2TuWt/cC0f50LmzJllPm/ePJm7ljDXkBGrDSVRpKS16tJL\nL5W52zeeeeaZyNtQvv32W5nHam8aMGBAXLYdL5999pnMP//8c7uMa3k4fPhwXMYUTyVLlpS5a6Gq\nVKmSzNevXx+3MZ1v6tevL/M+ffrYZVwTVGq3Vrk2Jtcc5cYZQghPPfWUzNPqGH3sscfsa67VzjVp\nuRaJU6dOyfzo0aPJjO7c+/nnn2XeuHFjmd91110y79q1q92Ga+Nx3Dk/KvceXKtKmzZt7LrcMXHy\n5MnoA4ugYMGCMv/pp5/sMlmzZpW5awOrUqWKzKO2VjmufSjWa+XKlZP58OHDZe7avUqXLi3ztWvX\n2jGlNvd5u+bTJ598UuauYcadx1Lizz//lHmXLl3isv5evXrZ1ypUqCBz99spXtzx49rdQgjhnnvu\nkfmOHTviMqZ4ypUrl8zddc6973vvvVfmsdot00rbtm1l7poTb731VpmnZWts4cKFZe6ul82aNZO5\na+lN7rWzxV/mAAAAAAAAJBAmcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACABMJkDgAAAAAAQAJh\nMgcAAAAAACCBJEQ1eayavgcffFDm999/v8xdjXG86gZdbergwYPtMrfccovM161bJ3NX3z137lyZ\nP/744zJPSVXx9OnTZV65cmWZu2pyV983YcKEyGNKFN26dbOvuWq/Tp06yTxqvWytWrVkfsMNN8jc\n1VaG4KtwO3ToIPNPPvkkmdGdnbJly8r8yiuvtMuUL19e5t27d5d5qVKlZO6OUbffHzx40I7JGT9+\nvMx///13mb/55psyd+emH374IdK///rrr2UeQgiLFy+2r0WRP39+mc+YMUPmbh9z+3cIqV977Lgq\n0s2bN8vc7d8hhPDKK6/IPK2qyTNlymRfy5s3r8zduczVJ7vKcnfsptVnEUIIlSpVkvmcOXNk7q6L\nOXLksNtIyTklClfR3LJlS5k/9dRTMr/66qvtNtx5w9W/btq0ya4rCncdcHXVIfiKYXc+cTXd7lg5\nfvy43Xa8uP2sT58+Mj9x4oTM3bVp5MiRMh8yZMgZjO7sfPPNNzJ354c777xT5q6eftGiRSkbWAQX\nXKD/O3ubNm1k7t5brGvcZZddJvN9+/YlM7qz07FjR5nfdNNNdhn3nTrz5s2TubsHP3DgQKT1x+Kq\nw91vJPc7LNb9a1rJly+fzIcNGyZzdy0YPXq0zP/yl7/IPF73lSGEUKZMGZkvWbJE5hkyZJD5iBEj\nZB7rfuOdd96R+enTp+0yZ4q/zAEAAAAAAEggTOYAAAAAAAAkECZzAAAAAAAAEgiTOQAAAAAAAAmE\nyRwAAAAAAIAEki4pKSkp1VaeLl1qrfqf3JPRixcvLvOuXbum5nDSVPPmzWXuGgfSp08feRuZM2eW\n+dGjRyOt58UXX5T5888/H3lM55sCBQrIfMGCBXaZq666KrWGE0IIIWvWrDI/fPhw5HW5p/K7p8FX\nqVJF5kuXLo287fNNgwYNZO7246pVq0behmuycfuZezL+xo0bZe72jdKlS8vctdiEEMKFF+qCxM6d\nO9tlosidO7fMX3jhBZnH+rzfffddmY8ZM0bm8WgciLdPP/1U5nffffc5Hsl/K1asmH3NNQdt375d\n5q41xrWBuVas87HlynHNna5NKAS/HyeSatWqybxfv34yv/XWW2Xu2lZuv/12mU+aNEnmR44ckXks\nF110kcxdO5BrJU2rlr2UcPeQrmEtS5YsqTmcFHFjcvfNQ4cOteuaNWtWXMbk2qweeughmbtmNHf8\nhBDCq6++KvOXXnopmdGdncmTJ8s81jXryy+/lPmKFStk3qNHD5m7hijXJJkS7p4wZ86cMs+TJ4/M\ne/fuLfNevXrJPNZ+mdoyZswo8z///FPmri330Ucflbnb72Nx1/enn35a5u484NoF3RyDa7kKIYTl\ny5fL3L1vR03b8Jc5AAAAAAAACYTJHAAAAAAAgATCZA4AAAAAAEACYTIHAAAAAAAggTCZAwAAAAAA\nkEB09Ugqc09qT0lbyLJly2Tuniie2urWrSvzeD3lPpYNGzbI3H3eKRG1tSpbtmwyL1SoUDyGc17a\nsWOHzP/xj3/YZVyzhWvCiMq1CfXt2zfyulwL1bBhw2Tumkf+E9qsvvjiC5m/9dZbMr/iiitkHqtN\n4cCBA5HyqFyj2eLFiyOv66OPPjrb4cS0d+9embv9251/QgihXbt2Ml+4cKHM27dvL/PVq1fL3DVw\nuPaKqVOnynz27NkyDyGETZs22ddS05NPPinzWMf0J598Epdtu1bA4cOHy7xNmzYyj9Vm5ZotUtJy\nFMW0adNk7traQvjPaLNy18xcuXJFWo+7H8yQIYPMBw0aJHPX/BeCb1n75ZdfZP7ZZ5/JPGprlWth\neeONNyKtJ56yZ88u89Q+TuLJjfW+++6Tubs+hODv84sUKSJztw/8/vvvMnfnOMe1ioUQwgcffCDz\nIUOGyDxe9xvuvBurRWvVqlUydy1AEydOlHm8Wqtcq28I/p7Qufzyy2WeSM3LrrXKyZs3r8xjtWFG\n5dZ1/fXXyzxW27CyZs0ambvGqlhjigf+MgcAAAAAACCBMJkDAAAAAACQQJjMAQAAAAAASCBM5gAA\nAAAAACQQJnMAAAAAAAASSJq0WbmWj507d8o81tOh3RO/P/zww+gDi4Pbb79d5pUqVbLLRG0UatKk\nicz/+te/yrx169aR1h9Lvnz5ZH7ixAmZjxkzRubdu3eP25jON5deeqnM3VPrQ0j9NqtrrrlG5q7R\nx+1LIYRw4403RlpX0aJFYw8uDZQtW1bmrh3o+PHjMnftVJdddpnM3TnuXHCNLhUqVJC5a9CpX7++\n3UaVKlUijalRo0Yyd213S5YskfnKlStlfujQIbtt12Sza9cumY8aNUrmVatWlfngwYNlPmnSJJm7\nZhPXLhJCCB07drSvpSa337/88st2mWeeeUbmrsEpaiNOzpw5Ze6O3VjcZ/7555/L/JtvvpG5a5Nx\nbVmvvvqqzF0jUkq4+wd3DMWrMa1atWr2tQkTJsi8d+/ekbbx888/y9x9b+4cEOt+rVu3bjJ357JY\n64rCXV+nT59ul1m7dm1ctu2u4a49zDX/nQuZMmWSuWuScffsd999t8xj3Rs5ruXRHROPPPKIzF1b\n4E033STzBx54wI7JnVNiXTOVli1byty1MxYoUCDS+kPw+36pUqVkfu+990behvL444/L3O1jIYQw\nY8YMmd9///0y79Kli8yHDh0aKY/KXWtCCKFgwYIy3717t8xLly4tc7d/u987DRs2tGOKauTIkTJ3\n++V1110nc3cf8uyzz8q8U6dOdkwlSpSwr50t/jIHAAAAAAAggTCZAwAAAAAAkECYzAEAAAAAAEgg\nTOYAAAAAAAAkECZzAAAAAAAAEki6pKSkpFRbebp0Mm/Tpo3M3ZOsXXNGCCGsWrVK5o8++qjMjx49\nateVmvr06WNfe/jhh2Xu2mdWrFghc/cE/I0bNyYzujM3b948me/Zs0fmrm3FNcakJdewVbFiRZm7\nVqeMGTPKPFbDw5QpU5IZ3dnJmjWrzMeOHStz1/AQQgh79+6VuWs1mDt3bjKjO/fc0/pffPFFmefP\nn1/mOXLkkPkTTzwhc9fGlBL9+/eXeePGjWWeO3dumbvziWsDcG1MIfhWO6dkyZIyv/XWW2Xu2rJc\nO1n69Onttt1+vGPHDpkPHDhQ5u4a5LiGB9egNGDAALuuiRMnRtp2arvjjjvsa+46586vUbkmo549\ne8p8wYIFdl2ubco1cl177bUyd/vfsWPHZO6uA65JLSXc/ufumVzbnbvOObGOE3d/FPXYuuAC/d8n\nXcNI8+bNZV64cGG7jU8//VTmrt3EtdRF5ZpQYrXbuHtn17KWPXt2mbtzorun2bBhgx2T49rAatas\nKfMiRYrI3L0310rjrsmuYS1qy14IIdSoUUPmbdu2lbk7/7j927UM/dd//Zcdk3t/UfdXd012x1zm\nzJllfvr0abuNt99+W+YLFy5MZnRnxzV6un01hBBWr14tc9csN3v2bJnH815Rueuuu+xrTZs2lbm7\nz4rKNWmdi/sZ99569Ogh81y5csl81qxZMnf7TAjxa7NV0zb8ZQ4AAAAAAEACYTIHAAAAAAAggTCZ\nAwAAAAAAkECYzAEAAAAAAEggTOYAAAAAAAAkkDRpswIAAAAAAEDyaLMCAAAAAABIcEzmAAAAAAAA\nJBAmcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACABHJhWg/gTHTv3t2+Vrp0aZlffvnlMt+5c6fM\nX3vtNZkvW7YsmdGlnvz588v81VdflXmBAgVkXrRoUZm7z/WTTz45g9GdWzfccIPMb7nlFrvMhRfq\n3full16Ky5iiypMnj33Njemhhx6Suds39uzZE31gQt68ee1rv/zyi8wPHTok81y5csn82LFjMi9f\nvrzMt27dasekuM8ohBCGDRsm82rVqkXahhvT3r17Zf7999/LvE+fPpG2G0/Zs2eXeZMmTWRev359\nu64NGzbIvFevXtEHFkH69OllPmPGDLuM28+mT58u83feeUfmS5cuTWZ0Z6ds2bL2NXfd2rdvn8zX\nrl0r882bN8t8/PjxsQcXBxkzZpT5b7/9JvMsWbLI3B1zGzdulHmdOnXOYHRnxr2H9u3by7xgwYIy\nL1OmjMzdZxGCP5etX7/eLhMPF110kczdPjZv3jy7rhdffFHmW7ZsiT4wYcKECTI/ePCgzE+dOmXX\nlSFDBplfeumlMn/33Xdl7s4z56M77rhD5n/++afM3Xd94sSJuI3JadmypcwbNWoUaT3uuujOMyGE\n0Lp160jbiJesWbPKvHr16nYZd/1z90B333139IHFSfHixWX+/vvvR1rP9u3bZe72mcKFC8s8X758\ndhvuu8iZM6fMy5UrF5f8448/lnmPHj1kHus+zn0ejntv7jfYP/7xD5kfP35c5keOHLHbbtu2bTKj\nOzPuvF6yZMlI+TXXXGO34c6jvXv3lvkXX3xh1/Xv+MscAAAAAACABMJkDgAAAAAAQAJhMgcAAAAA\nACCBMJkDAAAAAACQQJjMAQAAAAAASCBM5gAAAAAAACSQhKgmL1GihH3NVTq7OlJX1Tdp0iSZP/ro\nozKfNWuWHVO8uJpSVwnqqoRbtWol8549e8o8JdXkro7PfU7r1q2TeYMGDWTuqgBXr15txzRmzBj7\nmuJqNl1lnavidjXJrhYxhBB27dol8xYtWsg8XhXkzuWXX25fq1y5sszdMee42seVK1fK/JJLLom0\n/oYNG9rXGjduLPOKFSvKfMWKFZG27cbqapXjyZ0T3fcWlTtvhBDCsmXL4rKNeHGVxCGE8PDDD8vc\n1WyOGzdO5rGqKOOhWLFi9rWffvpJ5u58smnTJpm78/e54KqbXS36s88+K3N3/j4X3LaHDh0aaT25\ncuWS+cKFC+0yJ0+elPlTTz0VadtRue0+8MADMm/Tpo1dl6te79ixo8zdPuO0a9dO5lWqVJF5Sj5v\nd1/m6rjdfYVb/7nw2GOPyfytt96S+TPPPCPzl19+Weau9jol3P7UpUsXmU+cOFHm7j7EXctcBXRK\nXHXVVTJ31/B69erJ3N1z/vjjj3bb8+fPl3mfPn3sMmmlc+fOMt+5c6fMBwwYIHNXfe1MmTJF5u4e\nIYQQRo0aJfO//e1vMp8xY4bMP/zwQ5n/+uuvMj99+rQdkzJv3jz7mjvvut+kd955p8zXrl0baUzn\ngrsuPvfcczIfO3aszN09lrsnC8Hvl3/88Ydd5kzxlzkAAAAAAAAJhMkcAAAAAACABMJkDgAAAAAA\nQAJhMgcAAAAAACCBMJkDAAAAAACQQNIlJSUlpdrK06VLrVXH3fPPPy9z11TStGnT1BxOirin8k+b\nNk3mAwcOlHmsFijX2DBkyBCZ33rrrTKvW7euzF1Lk9uua5QKIfrT3d3T490T7ePZavCf7MILdWme\nO+YqVKggc/fEfMftMyGE8MEHH8j8+uuvl/mNN94oc9emcC4UL15c5osWLZL5FVdcIfODBw/Ga0gJ\nJWPGjDI/cOCAzF0b2Jo1a+IyHtcisn//fruMO4efj1xzy+TJk2Uer/a1tOTaMF3DpGtJufLKK+02\nrr32WpmvWrUqmdGdW5kyZbKvufuHkiVLyvzmm2+Oy5jiybUruWaYeDY7ReUaCRs1aiTzXr16ydy9\nN9eKtXXr1jMY3b8qUqSIzDdv3izz0aNHy3zDhg0yf/vtt2Ue694yKvddz507V+auJez999+X+eHD\nh1M2sDTg7svc/XcIvoW4fPnyMo9X26vbrmtaDCGEcuXKyTx37twyd9f3ffv2JTO6sxPr3vXuu++W\neayWv/PNLbfcIvOZM2fK3LUw3nvvvTJ3jb/ZsmWzY3LHe9euXWXuzkFq2oa/zAEAAAAAAEggTOYA\nAAAAAAAkECZzAAAAAAAAEgiTOQAAAAAAAAmEyRwAAAAAAIAEcl61Wbm2g9q1a9tlfv3110jbcO0S\nDRs2lLlrXVq9enWk7cbinu7et29fmbuWD9ccNXLkSJm3b98++cH9mxtuuEHm3333ncxnz54t8x07\ndsRtTPHiniju9o3ff/9d5pUqVZJ5/vz57bbdU+IHDx4s86+++squS3nyySdlfv/998s81pP0S5Qo\nIfOorT4PPvigzFPSeBHVBRfoeWzXbHHTTTfJvGbNmjLfvXt3ygYmuBaqdevWyXzSpEkyd0/fz5o1\nq8wHDRok81itE+ebtm3b2tdco9lvv/0mc/e+n3vuuegDE1xr3rZt2+wy7phbsmSJzFeuXCnzkydP\nJjO6s/fzzz/L3LV8uPe2a9cumbuGnrVr157B6FKHO8+kT59e5u57ePzxx+02KlasKHPXrhRVlixZ\nZO7uK1wDlbtuhBDCRx99JHN33frjjz/suuKhTJky9jV3nM6aNUvm7n279rr58+cnM7oz5/Yz1za1\nZcsWmbuGnhMnTsj86NGjMnf3M7HMmDFD5u6+2d1zli5dWubufOIapVLCNeW4lrC33npL5seOHZO5\na/b69ttv7Zjcd5fa3DV5+PDhdhn3Xbvm3c6dO8s8anOnawh2x0kIIXzxxRcyd9chd3+XOXPmSOt5\n8cUXZe6OxWbNmsk8hNjnasU1g7rf9GPHjpV5PBsYq1SpInPXAPrll1/K3P22Xb9+feQxufkH95ux\ncePGMqfNCgAAAAAAIMExmQMAAAAAAJBAmMwBAAAAAABIIEzmAAAAAAAAJBAmcwAAAAAAABLIedVm\ndfPNN8t83Lhxdhn3tP7Dhw/LvHjx4jJ/9913Zf7UU0/J/MiRI3ZMUV1yySWRtu2eou2eHO4aeooW\nLSrzWG1C7qnlrlFj0aJFMp88ebLMYz1hPbUVK1ZM5kOHDpW5a/v5+uuvZX7q1Cm77bJly8rcNWeM\nGDFC5q4Bze0bruEolnr16sn8tddek3n58uVlvnHjxsjbVvLmzSvzWMeoe8K/456+776366+/XubH\njx+PtN0QfEuKa6Hq3bu3zF2zhTv/uEYud84NIYTTp0/b19KCO1+FEMI//vEPmbvj1DXo5MyZM/rA\nInBtQiGEUKpUKZk3bdpU5q4Zsnr16jJPyf4aL24/c+eTpUuXyty9N9cgeD56+OGH7Wuu3bJDhw6p\nNZwQgm9hcdcHd20KIYRrrrlG5rGa3FKTa88JIYQMGTLI3F1vsmfPLnPXOOfuURPJe++9J/NY+7FT\nsGBBmbs2Ucc19LhWQ3feiKciRYrIvFq1anFZT6w2R3e/9te//jXSts8Fd//asWNHmbsG2ho1asRt\nTGnFNfy5e/m//OUvqTmcmHLkyCFz99vWzQG41tgQYv+uOt+45s49e/bI3M2h0GYFAAAAAACQ4JjM\nAQAAAAAASCBM5gAAAAAAACQQJnMAAAAAAAASCJM5AAAAAAAACSRN2qzc0/2PHTsm85MnT8ZtTO6p\n6L169ZK5e7r2TTfdJHP35P2dO3faMaV2A8zy5ctl3r9/f5l/+OGHdl2FChWSuWudcE+Pv/vuu2Xe\nuXNnu+3/jVxzhmtjuvDCC1NzODHlypVL5vv27ZN51LY7Z86cOTKP1ZbVqVOnSNvIli2bzBcsWCBz\n1yDnWj5SwjWuRG1bqVOnjszdecCd40KI77k6rQwfPlzm7thq165dag4nrtz+5xpdXBvh+cg1dbnG\nE3cNisU1OLljwrXMOO4807VrV7uMa6xx351r1HANMKNGjbLbjuL777+3r7lmyIkTJ8Zl22mpTZs2\nMm/QoIHM07LRMyrXXFagQAGZx2tfSgnXPDl+/HiZu4a1ROLuBULw7YzunOXaLdOS+z23f/9+mV98\n8cWpOZxzokWLFjJ3v1Wj3uumpa+++krmrk021mvu/Dp//nyZu+Mhntzv23vvvVfmrhWXNisAAAAA\nAIAEx2QOAAAAAABAAmEyBwAAAAAAIIEwmQMAAAAAAJBAmMwBAAAAAABIIGlSfePaZ2677TaZp6Qh\nJUuWLDIvVqyYzF2Tw/Tp0yNtd9q0aTKfMmWKXeaVV16R+SWXXCLz4sWLy/ypp56SufssZs6cacfk\nvP766zJftWqVzB9++GGZp6RJ5HzjWoDcE9lToly5cjLfs2dP3LYRL+6p+W7fiBd37MZqQnn11Vdl\n7lp9Nm3aJPOPP/5Y5m7fSEmbVf369WXunoDvmrRuuOEGmb/wwgsyr169uszj2Vjlzg/uvLt9+/ZI\n63fvIYRjNav/AAAgAElEQVQQJkyYIHO3Dzz//PORtt2qVavIY1JitYi4do4cOXLI/KGHHpJ5ly5d\nIo3pXLjssstk7vbjxx9/XOatW7eOvO3JkyfL3J13f/vtN5lXq1ZN5o0aNZL50qVLZe4aq0KI3jj2\n7bffytzdP5QuXVrmV155pczdvpQvXz47pqlTp9rXorjllltkPnfu3EjrcU2BIfh7F3c+dq2K99xz\nT6QxuWaiEydORFpPLOnTp5e5a4Rzn9PAgQMjb9tdk3/88UeZr127VuZ33nmnzLt16ybzp59++gxG\nd3YWL14sc9fu5Rp6Dh8+HGm77tgNIX7NTqNHj5b5p59+KnN3/jl16pTdhmvF7du3r8zdb8yo3G8w\nd+4Lwd/vxuu+yd1nR72Gxzofu0Yztx9HlTFjRpm7RspY1z+ncePGMm/evLnMH330UZkfOnQo0nYL\nFy5sX3O/9V1rVRT8ZQ4AAAAAAEACYTIHAAAAAAAggTCZAwAAAAAAkECYzAEAAAAAAEggTOYAAAAA\nAAAkkDRpsxo6dKjMXYPJkSNH7LquueYambsGkC+//FLmruUjalNEmzZtZN6vXz+7zLx582SeN29e\nma9fv17mrp3q/vvvl/nRo0ftmBz3pHb3ZPSaNWvKfPPmzZG3fb5xzTqzZs2Sea5cuey69u/fL/M8\nefLI/Pbbb09mdP9qzJgxMnf7WKyxuhaE2bNny7x8+fLJjO7suP3YtVqEEEKLFi1k7p4271rF3LnJ\nNcOkhGt5yJkzp8zfeecdmX/99dcyL1q0qMzPRWPaddddJ3PX9nP8+HGZu4aoWG2EVapUkfmuXbvs\nMlGULVtW5q6lqU+fPjJ/+eWX7TZck407B7k2R/e5OsOGDZN5iRIl7DJuGy5fuXKlzAcNGiRz10CV\nEq6dyn1+7hjat2+fzOvVqyfzb775RubxbJB78MEHZe7OWZUrV5a5uzcaMWKEzF27TQghnD592r4W\nhWsRdK157vwdq83KtVU2bdpU5vH67lwTYqw2K3dtdNeUgwcPyvyTTz6R+erVq+22o3KNY927d5e5\na5lxuWshmz9//hmM7uy4+w3XsOVamvbu3SvzbNmyyfznn3+2Y3L3ZbHaExX3/fTs2VPm7v4ke/bs\ndhvud8cbb7wh83Hjxtl1ReHO3+48E0L0e7+///3vMr/gAv23Fu7avmLFikjbjXWf497fXXfdJfML\nL9RTCVmzZpV5pUqVZO6apqK2mIYQQvv27WXesmVLmS9btkzm7j44c+bMMo/VyuY+1zVr1thlzhR/\nmQMAAAAAAJBAmMwBAAAAAABIIEzmAAAAAAAAJBAmcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACA\nBJIuKSkpKdVWni5daq0aAAAAAADgP56atuEvcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACABMJk\nDgAAAAAAQAJhMgcAAAAAACCBXJjWA0gtefLkkfmdd94p806dOsn8888/l3mfPn1SNjDhoosukvkN\nN9wg80ceeUTml19+ucw3b94s8/vuuy/5waWSypUry7xVq1Yyd99nuXLl7DZWrVol89atW8u8fv36\nMm/Tpo3Mjxw5IvMMGTLI/P3335d5CCHMnTvXvpYWLrzQnxrcvv/DDz/IvHfv3jJ/4403ZD5hwoRk\nRpd6KlSoIPM6derI3J1PNm7cKPOHHnpI5qdPn7ZjKlq0qMwLFCgg8/3798v84osvlvl1110n85tu\nuknmv/76q8xDCKF79+72tXioW7euzJs2bSrzIkWK2HXlzZtX5oUKFZJ5ly5dZD5+/HiZu3Pcd999\nJ3P3uf7+++8yD8GfUz7++GO7TDw8+eSTMm/Xrp1dJnfu3DI/evSozPPlyydzdz1bsWKFzF977TWZ\nr1y5UuYp4Y7FevXqybxUqVIyd/veggUL7LaHDh2azOjOTNu2bWVeu3Ztmbv7k0OHDsn8zTfftNse\nNWpUMqM7f2TKlEnmbj/Lnj27zDdt2iTzV155JWUDS0XuPb/33nsyd/czI0eOjNuY4uWTTz6Rubvm\nr1271q7LXUtTm7snO3XqlF3GnUePHz8u84kTJ0Ya06WXXirzYcOGyfztt9+263LXAnfededRd63p\n1auX3bbifrOFEML9998vc/d768SJEzK/+uqrZZ4jRw6Zf/PNNzJv3ry5zOMpa9asMnf3u+7e66qr\nrpJ5rHugJ554IpnRpY7ChQvL/OGHH7bLNGjQQObut6r77anwlzkAAAAAAAAJhMkcAAAAAACABMJk\nDgAAAAAAQAJhMgcAAAAAACCBJDuZs3Xr1lCjRo1QpkyZULZs2TBkyJAQQgj79u0LtWvXDsWLFw91\n6tQJf/zxR6oPFgAAAAAA4H+7dElJSUmx/sHOnTvDzp07Q4UKFcLhw4dD5cqVw2effRZGjRoV8uTJ\nE7p16xZee+21sH///tCvX79/XXm6dKk6+BBCyJIli8z/fSz/Y/fu3TKfM2eOzN2T3ZctW3YGo/u/\nXAtLCCH89a9/lfnChQtlPnDgQJm7J37/+OOPMi9YsKDM//zzT5mnRLVq1WQ+evRomY8bN07mixYt\nkvnf/vY3u+09e/bI/PDhwzJ3zUQzZsyQuWsKyJ8/v8wnT54s8xB8c1BaGTFihH3NNY+4/TWtdO3a\n1b7mmgXKli0r8+nTp8vcNQtUrFhR5m7fOHnypMxDCKFv374yf/bZZ2W+bds2mbsn5rvGPnf8uBaW\nEHyjULy89NJLMp8/f77M3WcRgm/zcJ+3O1+6Ji3XdvDUU0/J3LUypE+fXuYh+GOuQ4cOMl+6dKld\nVzy4VosQQsicObPM3X8Icu1rrtmwf//+Mr/mmmtkXqxYMZmHEELGjBll7prLqlatKvPZs2fL3DWa\nuWvvV199JfMQfHudM23aNJm7fcm1pPz2228ydw0z7rMLIYQ77rhD5r/88otdJjXFaqtxbYvuXtG1\nj7rWr/ORayZy55MWLVrIPFYrW2pzDTovvPCCzA8cOCBz13oagr/uR+X2v7Fjx8rctePFag/LlSuX\nzHft2iXzd999165LGTRokMxz5swZabshhLB+/XqZL1++XOapfR8yZswY+5prLPr0009l7lovV69e\nLXN3b+7u12K1r0UVr7G665y7Bn300Ud2TPH8vapUr15d5u6e07XjhRBCz549Ze7u8dasWSNzNW2T\n7F/m5M+f/591vVmzZg2lSpUK27dvD9OmTftnbVabNm3CZ599ltyqAAAAAAAAcJYiPTNn8+bNYfny\n5eG6664Lu3btCvny5QshhJAvX76Ys6oAAAAAAACIjzOezDl8+HBo0qRJGDx4cMiWLdu/vJYuXbpz\n8n+pAgAAAAAA+N/ujCZzTp48GZo0aRJat24dGjduHEL477/G2blzZwghhB07dtj/Ty0AAAAAAADi\nJ9nJnKSkpNCuXbtQunTp8Pjjj/8zb9iw4T8fYjt69Oh/TvIAAAAAAAAg9STbZjV//vxw8803h3Ll\nyv3z/0rVt2/fcO2114bmzZuHLVu2hCuuuCJMmjTp//eU8nj9X69ce0UIIXz44Ycyj9UelRaKFCli\nX3NNNq6dqnTp0jJ3n8X/Own3/5o7d64dU1StWrWSuXv6vmunci0f+/btk/n7779vxxS15SNe6tSp\nI/NZs2bZZf7nYeL/bubMmTJ3rWzxEusJ+G+++abMa9WqJfPrrrtO5q4FKGpTnNO6dWv7mmtimTdv\nnsxd05lrNnENJu54iMW1Z7iWtffee0/mO3bskPkFF0R6dFoYMmSIfc01KpyPXBtY9+7dZV6uXDmZ\nu30pKtfyEasFzzVB/f3vf5e5a02YMmWKzGOds1Kba8bq3LmzzHv16iXza6+9VuaugSME30jimt/c\nOd+1WaUltx+7truobrzxRpm7fSyEEIoXLy7z1L6Gu9Ygdy8VQgjNmjWTuWsOcu117thNS66JzDXx\nuGY01wh4Lrj7ELdfupY9d0y7PIT43Zfdc889Mnf3G66ZsWPHjnYbX375ZeRxKfXq1ZO5u3e9/PLL\nZb5169a4jOdciHWufOONN2Tu7v3ctcY19W7YsCGZ0Z2ZWPd97vjt1q2bzF0LrGtjOh+VLFlS5q6x\nb/DgwTJ3jVUhhHD69OnoAxPUtM2FyS1044032gG4Om8AAAAAAACkjmj/SRYAAAAAAABpiskcAAAA\nAACABMJkDgAAAAAAQAJhMgcAAAAAACCBJPsA5HMpe/bsMo/1hPiDBw/K/NixYzK/9957ZR6raSEe\nYjWeZMuWTebuieKuheW//uu/ZH7FFVfI3LUGHTp0SOaxuBYq9+R/9/1s2bJF5q5NaNSoUXZMjRs3\ntq/Fg2uv2LZtm8xbtGhh1+VaO77++muZjx49WuYDBgyw24hi5cqV9rW8efPKvGXLljJ37Smuleah\nhx5KZnRnJlYjiWuLcOcBty8988wzMk9Ja5XjGgFcO9Xtt98u859//lnm7vzqjjnXXhFCCDfddJPM\n//zzT7tMFK59Zvz48TK/66677Lrc5+GuQ4ULF5Z5vNqsXLtN7dq147L+EHyb1Q8//CDzc9Fm1ahR\nI5m/8MILMq9QoYLMXWtQrNYqxzVMOh988IHMd+3aJfN169bJ3LU2xro/cQ18rrgiamuVaz1ZvHix\nzCtWrCjzw4cP2224piHXHhaVO2+4RsBYbY4ZMmSQec2aNWU+ZswYmTds2NBuI7VVrlxZ5kuWLJG5\nu6a4a7hr4Nu0adMZjO7suPfmGkN37twp84EDB8o8tZtEQwhh4sSJkfJixYrJPFabXo8ePWTurqVO\nlSpVZO7uB925Mp7XudT2xBNP2NfcZ16wYEGZu3bYqNegqNy9QAi+ncodE64x2f2WjPX5pRXXYura\n7o4fPy5zd/0LwTdjxQN/mQMAAAAAAJBAmMwBAAAAAABIIEzmAAAAAAAAJBAmcwAAAAAAABIIkzkA\nAAAAAAAJJF1SUlJSqq08XbpI/949QbtDhw52GffkddcW4dqPUrv5KBb31Hf3tOw5c+bI3DU2uCeN\n16pVS+bVq1eXeVpyT7p3zV4h+PeXSNwT548cOSLzTJkypeZwUuTVV1+VuWvqcrnjjt233nrLLnPJ\nJZfIfMKECTJ3LTPNmjWT+fTp02XunuIfr7aneOrSpYvMBw8ebJfp16+fzF1TQFSuGa19+/Yyf/31\n1+26XOucOx936tRJ5q7B63zkGspc0959992XmsNJEdcm5BoVXQNMrP3YNTjVrVtX5q6N8OjRozJ3\n1+onn3xS5m6fDCGEqlWrytw1E8VLnjx5ZL5nzx6ZlylTxq7LnS8HDRoUKXeqVasm85tvvlnm/fv3\nt+tyLWGO2y/ddcu12rl9xh0PIfj7BHe8u/a1b7/9Vubvv/++zF1DT6yWsLTSs2dPmbsWyZQ037rm\nsmnTpkVeVxTuGA3BtznmyJEjtYYTQvD7ntvvQwjhyy+/TK3hxF2NGjVk7n57umuHu1a7+5bzUe/e\nvWXuGs3i+d6iHnOuVdp9n23btpV5rN+d7rrSp08fu4yipm34yxwAAAAAAIAEwmQOAAAAAABAAmEy\nBwAAAAAAIIEwmQMAAAAAAJBAmMwBAAAAAABIIBem9QD+X6tXr5Z5586dI6/LPcV99+7dkdeV2lwj\nV7wULVpU5il5kv79998v81atWsncNVu49hnXMuSefv7YY4/J/FxwTzNfvny5zFesWBF5G+7zyJgx\nY6T1uCesb9y4UeaxjpOmTZvKfP78+TKvVKmSzHv06GG3EYV7bwsWLLDLuBa0zZs3R9r2iBEjZL54\n8WKZb9q0SeYDBw6MtN14cs09v/32W+R1VaxY8WyHE9OqVatk7pq3UuLUqVORcueuu+6SuWuMWbhw\nYaT1x1K8eHGZv/LKKzJ3rT7noxMnTsh86dKlMi9cuHDkbbjGopkzZ0ZeVxSuzapevXp2mSZNmsg8\ntdus3LXdce1AIfjmkbFjx8rctVm1a9dO5iNHjpT5okWL7JjiJVeuXDJ3TZXOhx9+KPNY90Cuzcq1\nNjpFihSR+VVXXSXzlLRWuXu8Bg0ayHzq1Kkynzx5sszd9cw1xbnmxJS49957ZZ7abVauhTMEf92P\nyrWpufO0u4afjy26KeGa39yx8tprr8ncNcI1b948ZQP7N/ny5bOvRb3XcU3U7rzet2/fSOtPicqV\nK8vcHXOHDh2K9O9d/txzz9kxpeZ9Fn+ZAwAAAAAAkECYzAEAAAAAAEggTOYAAAAAAAAkECZzAAAA\nAAAAEgiTOQAAAAAAAAmEyRwAAAAAAIAEkibV5HPmzJG5qxIePny4XZerUO7Zs6fMv/vuu9iDO0tt\n2rSJvIyr33T1qK7quVevXjJ3ldiuEi8WVwHs6qo7dOgg8yFDhsj87rvvlrmrknb1lPG0bNkymb/6\n6qsyT0kFuasa//TTT2XerFmzSOt3x9y6detkvn//fruuYcOGydxVp7rq8Hh54oknZP7WW2/ZZVq1\naiXzUqVKydxVxLtj9OjRozJv3LixzGNVk7saR3eu+f3332U+Y8YMmd9zzz0yd/XWrlo0BH+cRpUl\nSxaZlylTRuau/te9h1jbcPXQr7/+ul2X4o7dEiVKRFqPq34NIYR+/frJ/IYbbpD5bbfdJvM//vgj\n0pjSUrVq1WTuKoavuOKKyNtw1zm37Z9++knm27dvl7k7z7jrX6zvx11L46V9+/YyP3jwoMxdFbyr\nig3Bf3eu0tnJli2bzCdNmiTz3r17y3z9+vWRthtCCA888IDMN2zYIPO9e/dGWr/7vGvWrGmXmT17\ndqRtuxpwd41t0aKF3XZU7hob9V6na9euMm/UqJHMr7vuOplHrWeOxV2T3T2Ku+a79bjrYv/+/e2Y\nxo8fb1+Lwn0/EyZMkHn37t1lnpLfI6mtT58+Ms+aNatdxt1XfPbZZzKfNWuWzD///HOZuxrw1atX\n2zEp7jdBCCGULl060rr27dsn83r16sn85MmTkdafEvPmzZO5+73t9lf3fV5zzTUyd3MPIYRQt25d\n+9rZ4i9zAAAAAAAAEgiTOQAAAAAAAAmEyRwAAAAAAIAEwmQOAAAAAABAAmEyBwAAAAAAIIGkS0pK\nSkq1ladLJ/N8+fLJ3LUmdOrUyW7j2LFjMt+4caPM3RPt4/V0bdci8sgjj9hlXNvGkSNHZO4avIYO\nHSrzH374wW47KtcC1LZt20jrcU9Sd41IW7ZsibT+eHJPjx8wYIDM//zzz8jbcA03y5cvl3nUFgm3\njw0aNEjm1atXt+v6+OOPZd6xY8dIY4qXCy7Qc9IvvviiXaZ8+fIyL1iwoMzXrl0rc9cg4Bqf3L/f\ntm2bzEPw312NGjVk7hqcrr32WpkvWrRI5iNGjJD55s2bZZ4Sru3OnWe2bt0q8127dsn8+PHjkce0\ncOFCmbt2Cce1hLkGKve5uoaeEEIYPHiwzF1jQ1rKkSOHzN11v0KFCjJ3n5NrQzl8+HDyg/s37hiq\nXbu2zN01oly5cjJ3LYKuFWTcuHEyDyF2u1w83H///TJ3rTSLFy+WubuWhRDCBx98IHPX4BSV25dc\nS1Pu3LnturJnzy5z15T6/PPPy9w1IToXXXSRzJs0aWKXiXpNnj9/vsxfeOEFmcezlca1WbnW0Msu\nu0zmmzZtkrlre03J+SFeXCPlww8/LPOiRYvK3B0nrhUrhBBGjRqVzOjOjLuHdL+1/vrXv8o81v3a\n+ca1GoYQwsUXXyzz22+/XebuPs41dLpGJLffO+5+MIQQ3n33XZl///33MnetX+6+LC25RlnX5Oeu\n4e5cOXLkSLvtNWvWJDO6M6OmbfjLHAAAAAAAgATCZA4AAAAAAEACYTIHAAAAAAAggTCZAwAAAAAA\nkECYzAEAAAAAAEggadJmBQAAAAAAgOTRZgUAAAAAAJDgmMwBAAAAAABIIEzmAAAAAAAAJBAmcwAA\nAAAAABIIkzkAAAAAAAAJ5MK02OiAAQNkXqtWLZlnyJDBruvYsWMyX7t2rcw//fRTmU+dOlXmp0+f\nttuOl0suuUTmjRo1kvmNN94oc/c5ValSReb9+vWT+ejRo2WeEoULF5Z506ZNZX777bfL/LbbbpP5\n5s2b7bYnTZok82eeecYuk1ZatGgh86eeekrmr732mswnT54cabsXXKDnc8uVK2eXueOOO2R+ww03\nyLxQoUIyP3nypMwbN24s823btsk8V65cMm/SpInMQwjh0Ucfta8pb775pszHjh0baT3nQr58+WR+\nzz33yDxPnjwyz58/v8zdMR2CPx47duxol0lN6dOnt69Vq1ZN5p07d5b53r17ZR51X3IKFCgg8/bt\n29tlLr/8cpkXKVJE5u5YWbBggcwff/xxu+2olixZIvOLLrpI5lu2bJG528fmz58v84kTJyY/uFTy\n7LPPytxd/w4fPizz5cuX2224+yl3vnSuuOIKmXfp0kXmtWvXlvnFF18s8927d9ttt2rVSuY///yz\nXeZ8U69ePZm749dd/1q2bCnzZs2ayfzJJ5+0Y3LbKFasmMxnz54t85w5c8p8586dMu/fv7/MN27c\nKPN4qlu3rszbtWsXl/W7zzSEEO67775I68qdO7fM//KXv0RaT4UKFSL9+xD89eadd96R+ahRoyJv\nI17cvu++U7d/9+zZU+YTJkxI2cBSkbu2P//88zIfOHCgzFevXh15227fcN+Du565a9CQIUNkvmjR\nojMY3dlZtmyZzHfs2CFz9/v8xx9/lPnw4cPtttesWZPM6FKOv8wBAAAAAABIIEzmAAAAAAAAJBAm\ncwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACABJIuKSkpKdVWni6dzF2jxv79+yP9+xBCKFWqlMxd\ns9OgQYNk7p4c7p5kHU8vvfSSzF2zgGseOXXqlMxnzJgh8zfeeEPm3377rcxjKViwoMxdI8mVV14p\n861bt8r80ksvjZSHEMLVV18t8ylTpthlUlOsVjb31Hf3hHr3Xb/++usydw0ZvXv3lvkvv/wi8xBC\nWLhwYaQ8R44cMv/kk09knj17dpkfOXJE5jNnzpR5rPaFH374Qea//fabXSatuGYi1yqVNWtWmbdu\n3VrmefPmjTQet++F4NvX4vUUf9ea4BrT3PETgj/m3Htw540TJ07YbSiuKeKLL76QuWuuCyGE6dOn\ny9wdQ9OmTZN5jx49ZO6uQSnhmvMuu+wymZctW1bmrnXJNZu4Bp17771X5vFUo0YNmXfv3l3mq1at\nkrlrqAvBNw+WL19e5q6Zr2HDhjJ3zUQff/yxzPfs2SNz1zIUgm/QdI16qc3dz4TgryvuHnLMmDEy\nHz9+vMxdW5trLLr55ptlHoK/nh0/flzms2bNsutS6tevL/M+ffrIvGrVqpHWH4trex0xYoTM3Xt2\nrWyufS1Wm9WhQ4dk7po+Bw8eLHN3/t6wYYPM3f2aO9ZDCOHpp5+W+U033STzlDRmxYs75p544gmZ\nlyhRQubff/+9zN1v1XPBjfWhhx6SuTvvZsqUSebfffedzBs0aGDH5LbhmrFcs5M752fOnFnm8Tzf\nd+jQQebvvvuuzN1cgru/c+cNd20Pwd9bus/bUdM2/GUOAAAAAABAAmEyBwAAAAAAIIEwmQMAAAAA\nAJBAmMwBAAAAAABIIEzmAAAAAAAAJJA0abOKKlazwH333SfzJ598UuYTJkyQ+bhx42S+ZMmSZEZ3\n/nBNRn/++afMXZvVwYMHI2/7vffek/kdd9wh80WLFsm8b9++Mk+k78GZM2eOfS1nzpwydy0IURt0\nXNtP0aJFZe7ankLwrRCuMeujjz6SuWtXcsei4xrQ3HsOIYR169bJfMWKFTL/7LPPZD5p0qRkRnf2\nunbtKnPXcuTa/1wbWCJ59dVXZe7amP4TuFanEEKoVKmSzN9//32ZL1u2TOZNmjSR+fbt25MZ3fnD\ntUhUq1ZN5q4NJyWyZMkic9d44Zrodu3aJfNu3brZbbtz9aOPPhpp2wcOHJB5Su4HlIsuusi+5q5n\nriHKnfPjpVixYvY119xy/fXXy3z9+vVxGVNauuWWW2Tums5efvllmU+dOjXytl3bj/tcr7nmGpm7\nJp5zwTX2xev86lpdY13zXYOca9KqU6dO9IHFiWs5co1F7lh84YUXZO5+j6RE7ty5Zf7888/LfPny\n5TJ338/p06dTNrB/M3bsWPuau565NlHX+uyOOXet6dKli8yjtuyFEMKQIUNk7vaBffv2Rd6GEqt9\n1LXODR8+PNI2aLMCAAAAAABIcEzmAAAAAAAAJBAmcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACA\nBJIQbVbly5e3r/Xu3VvmrsHCtSaMGTNG5q5dyTX6nAtuTJ06dZL5zJkzZX7PPffIPCXfW5UqVWTu\nmjM2bNgg8wEDBsi8fv36kceU2tKnTy9z93T6vHnz2nX9+uuvMnetGq69IF7KlCljX3P7n/uumzVr\nJvNNmzZFH1icZMiQQeY1a9aUefv27WVevXp1mbv91bVlxXLBBXrO3bURuG27Y9Q9YT8tde7cWeau\nzcodD8eOHYvbmFLbY489JvO33nrLLjN+/HiZv/322zI/deqUzF2z4c0332y3rcRqLOrXr5/Mb7vt\nNpkXL15c5pkyZZL57t27ZV6uXDmZ79y5U+Yp4c6XrskvX758kfJYTTKzZ89OZnSpw30PrvkoVsvH\n9OnTZf7cc89FH1gq+/bbb2X+888/y9xdO+LFteeE4Nteb7zxRpnv3btX5jNmzJD5woULI60nnqZN\nmyZz1wz64IMPyjwt70OiWrBggczdfaJrpgrBNzi1atVK5mnZbOjuXb766iuZu5bW/fv3p+p4Qgjh\n4Ycflrk7bzRo0EDmw4YNk/n8+fOTGd2Zcc1UIfjfI0uXLpV5jhw5ZO6u+e73eYUKFWQ+cuRImYcQ\nv/NryZIlZe5+Jz/yyCMynzx5st2Ga+uKijYrAAAAAACABMdkDgAAAAAAQAJhMgcAAAAAACCBMJkD\nAG7h6fcAACAASURBVAAAAACQQJjMAQAAAAAASCBM5gAAAAAAACSQhKgmjydXndqtWzeZu7rvyy67\nLG5jihdX4T569GiZu5q0efPm2W24Gsx4VVG6CjpX9+eqdlPi4osvlrmri926davM27ZtK/P3338/\n8pjc4enqNw8cOBB5G0rdunXta127dpX5vn37ZO5qIk+cOCFzV/m3Zs0aO6a0ctVVV8l85syZMnf7\nhqt4TYn33ntP5gUKFJD5nXfeGbdtx0vDhg1lnj59+kjrmTJlSjyG8x8jS5YsMneVt64q1MmWLZt9\nzdWf79ixI9I23Hlg4MCBMi9cuHCk9Z8LF1yg/zvayZMnZZ4rVy67roMHD8ZlTFE99dRTMu/Ro4fM\n3fk+hBB69eol8xEjRkQfWCpz94ru83D3iu67jsrtSyGEULZsWZlnzpxZ5s8++6zMjxw5InNXfZ4S\nbkxHjx6NtJ677rpL5i+++KLM169fL/POnTvLfNeuXZHGE0+uUt5VQ9erV8+uy/22KVOmjMzd55SW\n3L2o464Racndl7n91f0WSiSujv3zzz+X+aBBg+y6ou4DTsuWLWXufv+5761SpUp2Gxs3bpS5u3Yc\nO3ZM5lSTAwAAAAAAJDgmcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACABMJkDgAAAAAAQAJJkzYr\n18ywf//+1BpKsho1aiTzAQMGyLxYsWKpOZwQQgjZs2eX+dChQ2Xu2iLcU/mjtgSE4J8q3r17d5m7\np3E77mnm56Jx55dffpF5uXLlZH748OG4bdu1Tvz0008yz5gxo8xjNYaklUyZMsl8/vz5Mnef6623\n3hqvIaW6PHnyyPyHH36QuWv8CsG3lZw+fVrmtWvXlvnYsWNlnj9/frvt1OZa02bNmiVz11Ry6NAh\nmc+ZMydlA0tF7ng4fvx43LbhWqU++eQTmU+aNEnmroHPtULGq6EnBL/ff/vttzJftWqVzN31L5b+\n/fvL/Pnnn5e5a/uJyt2OXX311XaZTZs2xWXbqa148eL2Nde4uXbtWpm3a9cuLmNKCdeOtmXLFpk/\n88wzMn/99dfjNqbU5lpDr7zySpm7zyKWMWPGyPzJJ5+U+e7duyOt352zHnvsMZl37NhR5rEa/lJy\nT51WXAve8OHDZf7000+n5nBicvdH7tznWhvj9f24fSnWtmvUqCFz99tz7ty5Mh81alQyozt/VK5c\nWeafffaZzH/88UeZt2rVym4j6ndav359mW/YsEHm7ndhSjRt2lTmd9xxh8xd+y1tVgAAAAAAAAmO\nyRwAAAAAAIAEwmQOAAAAAABAAmEyBwAAAAAAIIEwmQMAAAAAAJBALkyLjboGE/dE5xdeeMGu69df\nf5V5hgwZZN6yZUuZf/DBBzJ3Y3Vck8yff/5pl3nwwQdlXrBgQZm7p8qvXLkymdGdPfddfPrppzJ3\nTSLVq1eXecmSJVM0rnj48MMPZe4aiNxT0Xfu3Cnzu+++2257xIgRMnf7a9TWKtcM4/bvvXv32nX9\n7W9/k7lrvGjTpo3MK1asKHPXHhZV5syZ7Wvuu3NtP1HVrFkzLusJIYSpU6fKfNu2bTJ/5JFHZP7l\nl1/GbUzx4s75bv9zrXk33nhj3MYULx9//LHMv/vuO5m//fbbMs+dO7fdRr169WT+6quvyvyNN96Q\nuWutctz1skOHDnYZ13Tlvju3jXXr1sn8iSeesNuOyrWpffPNNzJv3ry5zN152rVCLl26VObxbNRI\nK9u3b7evucanqOdjt8+4a7s7tlzjXAj+2uH27z/++MOu63xTpUoVmbv7B5enxJQpU2TujomRI0fK\n3O0zv//+u8xnzJghc3etifWbwL0Hx7XVdOrUSeY9evSQ+aJFi2Qe67eTazx866237DJK1EZKd1/m\n7nVDCOHNN9+Uefny5WUer9aqxYsXy9w1H4Xgm46rVasmc3fuGz9+fDKjSz0FChSQuWt+K1SokMzd\nb55x48bJ3P22jWdLnPv9Mnv2bJn37NlT5u68ceDAAbvt6dOny/wvf/mLXeZM8Zc5AAAAAAAACYTJ\nHAAAAAAAgATCZA4AAAAAAEACYTIHAAAAAAAggTCZAwAAAAAAkEDSpM1qwoQJMl+2bJnMX375Zbuu\nsmXLynzXrl0y//7772VeokQJmW/cuNFuWxkyZIjMYzVNuadlb9q0KdK2zwXXzDBgwACZz507V+bu\nSfquKeJc6NWrl8zde16/fr3MDx8+LPMvvvjCbvuaa66R+ebNm+0yUWTNmlXm7un0ZcqUsetyrVUu\nd/tA5cqVZb5mzRq77ShiPQF/3759MndjdYoXLy7zn376SeauLSSWF198UebDhg2T+dixY2XesWPH\nyNtObe6YW716tcxd+0fUdrdzIWpjg2sqccdVCCF89tlnMq9Vq5bM43VNmTx5ssxd818IIVx88cUy\nX758ucxdo4trSTl9+rTddlSu2SJPnjwyd+d212LpWrEeeughmcfzvcVLkyZNZP7uu+/KPNZ+7Jqu\nbrvttkhjco19rpnItZ7Onz/fbuOrr76SedeuXWXu2i3PhaJFi8q8Xbt2MnctjA0bNpR5vO5PQvBN\nUEuWLJH5448/LvPevXvLvEKFCjI/cuSIzF955RWZu3bJlBg1apTMXZPR4MGDZe5aCmOdj10DUazW\nOaVq1aoydy1X7jhxvwtDCKFw4cIyP3jwYDKjOzuukdLdr4cQwrx582TeoEEDmcdqjk0rx48fl7n7\nXbBixQqZlypVSuauifVccPdAbi7BNcK5lt4cOXLYbbsmt27dutllzhR/mQMAAAAAAJBAmMwBAAAA\nAABIIEzmAAAAAAAAJBAmcwAAAAAAABIIkzkAAAAAAAAJJF1SUlJSqq08XbrUWjUAAAAAAMB/PDVt\nw1/mAAAAAAAAJBAmcwAAAAAAABIIkzkAAAAAAAAJhMkcAAAAAACABMJkDgAAAAAAQAK5MK0HcLZu\nu+02mQ8cOFDmWbNmlfm7774r88GDB8v8+PHjZzC6M5MtWzaZP/zwwzKvWbOmzCtVqiTzhQsXyvzF\nF1+U+apVq2R+LhQoUEDmDzzwgMxz5cpl13XJJZfIvH379pFyty+NGjVK5nPnzpX5VVddJfMQQuja\ntavMW7duLfOvv/7arkt55ZVXZO72pQwZMth1ffPNNzIfPXq0zFevXp3M6M4fmTJlknnnzp1lfv/9\n98v88OHDkbZ7/fXXR/r3IYSQO3dumT///POR/v3VV18t8yxZssj81KlTdkzunLVo0SK7TGpKnz69\nfa13794yv+OOO2Q+ffp0mbvPO17ceSwEfw6/9dZbZe6O60GDBsl82LBhsQeXiu655x6Z79q1S+bf\nffedzDNnzizzo0ePpmhc8VC8eHGZu/NMvXr17Lrc+37ooYcijyuKpk2byrxjx44yz549e+RtHDx4\nUOa1atWKvK54admypczd/YO7n3r//fdlvmbNmkjjcfe0IYTwxhtvyDxjxowy//XXX2U+depUmafl\nvWJqc/txjRo17DLNmzePy7avuOIKmbdr107m7t6yWLFidhsbNmyQ+csvvyzzjRs32nVF4c7Hw4cP\nt8u4c74bk/v3bj+OKtb5x90DFS5cWOaHDh2SubuvOBf3UmXKlJH5J598InN3nxX1GuvuRd1vrRBC\nePTRRyNtIyr3m7RLly4yj3W/1q1bN5kfOHAg+sD+DX+ZAwAAAAAAkECYzAEAAAAAAEggTOYAAAAA\nAAAkECZzAAAAAAAAEgiTOQAAAAAAAAkkXVJSUlKqrTxdutRadYqVLVtW5rNmzZL5pk2bZH7LLbfE\nbUz58+eXuXtq/rp162T+xx9/yLxt27Yyb9GihcxLlCgh83i64AI9jzh06FCZ58mTR+auESIE/zlt\n2bJF5u7J+OXKlZO5awtxn6truQohhPfee0/mmzdvtstEUa1aNZmn5Mn49evXl7lrn3GNWVWrVpW5\nayKIyjVThRDC66+/LvPatWvL3DUfTZkyReaulca953vvvVfmsbi2Kfddu/YK1061Z88emVeuXNmO\nyR2/sZaJB/d5uzwE3/D2ww8/yHzp0qUyd99DvMRqR2jUqJHM3bmsSZMmMnfnV9fWFk+u2WnBggUy\n//zzz2Xuzutu/37ppZfOYHRnxzXiuJbMiRMnyvzDDz+023DNhidPnkxmdGemcePGMn/wwQdl3rBh\nw7hs91xw+577HkII4bfffpO5+7wLFSok8x49esj822+/tduOyrV9XnvttTJv0KCBzO+77z6Zu6bP\nc3FsReVaY11LoWuUuvPOO+02Vq5cGXlcSrNmzWTuWjJ/+uknmW/bts1uwzXr7NixQ+Zjx46164qH\nWPcIderUkXnp0qVl7r4jd1x36NBB5kWLFpW5OyeG4Jsh3XXItdGNHDlS5u76N27cODsmJWfOnPa1\n/fv3y7xkyZIyd/eW7neya9V0v5FOnDgh85RwbWrz5s2TecWKFWXufsPG+h5c8+Tvv/8u85kzZ8pc\nTdvwlzkAAAAAAAAJhMkcAAAAAACABMJkDgAAAAAAQAJhMgcAAAAAACCBMJkDAAAAAACQQJjMAQAA\nAAAASCAJX01esGBBmXfv3l3mtWrVkvnevXtl3rRpU5nHqz45losuukjmZcqUkbmr17vhhhtk3qZN\nG5mvWLHiDEZ3doYPHy5zN6YhQ4bI3NXXhhDC1KlTI41p9+7dMnf7kquZc3WQx48fjzSe85XbL/v3\n7y/zLl26yPzpp5+W+eLFi2XuqgOrVKki8++//17mIYRw++23y3z27Nl2GeWDDz6QeY0aNWTuKkeX\nLFkSabvngqvcdrXuIYSwevVqmQ8bNiwuY3Jc1WWsSsvbbrtN5s8884zM27VrJ3NXiZ2W7rrrLpm7\nOnZ3PGzevDleQwoFChSQuTuHjxgxQuZ9+/aN25jiJXfu3DLfs2ePzF0N77Jly+I2pqhcTbernb3k\nkktk7mpt/1NkzJhR5u3bt5d5165dZd6qVSuZL1q0KGUDiwNXxTxq1CiZuyruzp07x21MUdWuXVvm\n7733nsxd1fOzzz4r86NHj6ZsYKnI1ST37NnTLlO2bFmZt27dWuZ//vln9IEJuXLlkvlVV11ll6la\ntarMXUV43bp1ZX799dfLPC2POSdHjhwyd/dYhQsXjrR+9/slhBB+/fVXma9Zs0bmR44ckbmr1o7n\nfUVU7vdLy5YtZe7u79zvxUKFCtltT5w4Uebud4f7vUA1OQAAAAAAQIJjMgcAAAAAACCBMJkDAAAA\nAACQQM5oMufUqVOhYsWK4c477wwhhLBv375Qu3btULx48VCnTp3wxx9/pOogAQAAAAAA8N/OaDJn\n8ODBoXTp0v98oHG/fv1C7dq1w8aNG0OtWrVCv379UnWQAAAAAAAA+G8XJvcPtm3bFmbMmBGee+65\n8Oabb4YQQpg2bVqYO3duCOG/24duvfXWVJ3Qcc0wIYT/w96dB9xUPur/v0WRIRkyhYhIihARRWRs\nIEMqRcgYSUpFpZSpklL6NMkQpTQpEcoQmgkpQzJEIfMYIb8/vr9zvudzvte1WU9re+xz3q8/r22t\n+97Ps/Zaa989rSuMGTNG5q7d5KeffpL5Y489JvMT0Vrlnir/2WefyXzTpk0yL1eunMzd0+PT8y+q\ntm/fLnPXlLNq1SqZu8aBEKK3WZUpU0bmro3p+eefj7R/1+YSgm9lu/HGG2Xujo24NGvWzL729ttv\ny9x9trp06SJz98T8ESNGyPyiiy6SuXsyfvHixWUegm8cc0+0d21q7in0AwcOlPm3334r8zib/6pX\nry7z6667TuauWcc1/7m2ixBCGDdu3DFmlxyzZs2SuWvkCiGE8uXLy9y11SRqxkqmROcN1wr4H39F\n+9+98MILMj98+HD0iUXk7hHc59Qdr+7z/t5778k8rhaWRFwbpmsMefPNN2Xeo0cPmX/yySdpmpdy\nzjnnyNydv901qGrVqjJ3n7lEDSbp1ebn2kpdI1IIITRt2lTmefPmlbm7drhWmqgaN25sX3OfOXdP\nmD9/fpm7+2OXnwju/O0+KwsWLJD5hAkTZJ6erVXuGuTumVyznNtPCCF8//33Mi9QoIDM161bZ/cV\nRe/evWWe6F7e3Te533WvXr1k7tqY4tSxY0eZuzm5a4dr3nLX9qjcNSgE3x7mjr8cOXLIfPTo0TJ3\nn62bb75Z5nF+Vz106JDMXYuuO4deccUVMnctZCH45jzXhBjFMf8yp2fPnuHJJ5/8t+q7zZs3/+dJ\nP3/+/CdkwQMAAAAAAADHWMyZPHlyyJcvX6hQoYLsNQ/h//wX5Tj/qzIAAAAAAAC8hP+b1RdffBE+\n/PDDMGXKlHDgwIGwe/fucOutt4b8+fOHTZs2hQIFCoSNGzeGfPnynaj5AgAAAAAA/K+W8C9zBg4c\nGNavXx/WrFkTJkyYEGrXrh1ef/31cN111/3ns2rGjBkTmjRpckImCwAAAAAA8L/dcbVZ/Yf/+N+p\n7r///jBjxoxQqlSpMHPmzIQPrgIAAAAAAEB8Mhx1D8OJY+cn4bN0SpQoIfNly5bJ3LUMbd26NbY5\nxeWrr76S+dChQ2U+ceLEZE4nVlmzZpX5l19+abepUqWKzE9Eu0lU2bNnl/nKlStl3rZtW5lPmzYt\n0rgPPPCAzGvXrm23eeihh2Tujj/HtaG4/V999dWR9n8iZMqk/0/VH374QeYdOnSQ+bx582KbU9Gi\nRWUe9Vx2xhlnyNwdMyGEMGTIEJknu60mY8aMMj9y5Ijdxs3JtSfu2bMn0pxcS5hrVXGKFStmX2vd\nunWkfbm2mooVK8rcNWq41rdE723FihUyd8efa6p0c3JtK64hIz1deumlMncNid26dbP7+vrrryON\nPXfuXJlnyZJF5tOnT5e5a7/avXu3zF3LXgi+Mat06dIyj+sa7lqr9u3bZ7dxTYiFCxeWuWuzcs06\nbdq0sWMr7t4hBP/+XPuM+/eucXXAgAEyd60tcd57/ddSlv/qzDPPlLn7zE2ZMkXmZcuWlbk77k8E\n15TjGnpcy1UIIbRr107m7vzasGHDY8zu+LhrkGvpDcHfj7oWy5OR+91FbdF1x7druUp0D5Re2rdv\nL3P3nl1zXZxcw59rrXJthPPnz7djuKbeqK2Aatkm0l/mAAAAAAAAIH2xmAMAAAAAAJBCWMwBAAAA\nAABIISzmAAAAAAAApBAWcwAAAAAAAFJIurRZZcuWTeaJGgScU089VeYNGjSQ+bBhw2S+ceNGmdes\nWVPmf//9t8yvuuoqmX/66acyT4tLLrlE5q6l4pxzzpH5H3/8EXns119/XebPPvuszNeuXStz93sr\nXry4zN3TvidNmiTzEHyLhOPaPC644AKZL1y4MNL+08LNybVcuSaj9OQa5N58802Z169fX+Y7duyI\nbU5R9ezZU+au+aFPnz4yX7VqVWxzSi+JPleuTe3jjz9O1nTSzDWOufaCqMff+++/L/P33ntP5u7c\nGifXIOdayNz1z72HW265xY7tPteuBc9dkx33+3QtH+7alJ7cfUurVq3sNrfeemuypnPCuHs/d+1I\n1HyTXjJnzizzRYsWyXz27Nky79KlS6T9n4h2znLlysncXQtcc12nTp1im1Ncxo8fL/Nff/1V5ona\nHP8ncPe1rvHw1VdflblrCH7ttddknuh8v2bNGpl/8MEHMu/bt6/M3fc2J1++fDJPy3enuLgmqDp1\n6sg8zjZH1xrq8r/++ivS/t2x17RpU7uNu467BmRn//79MnffVV2T37333mvHqFevnsxnzJhxjNn9\nO9qsAAAAAAAAUhyLOQAAAAAAACmExRwAAAAAAIAUwmIOAAAAAABACmExBwAAAAAAIIVkSo9BH3nk\nEZmPGDFC5h06dLD7cq0x7gnXzZo1k3lczUQdO3aU+Zdffmm3OeUUvabWpEkTmb/88ssyb9eunczj\nfPL65MmTZf7tt9/K3D093r1n90TxO++8U+YjR46UeZzcE/OvueYamS9ZsiS2sTNl0h/R008/PbYx\n4uJa1p577jmZu5aZqK1B7unxr7zyit3m+uuvl7k7nzzxxBMyv+GGG44xu3+uYcOGMp8/f77Md+/e\nHWn/7vd24403yjzR72fq1KmRxnYtCEeOHIm0H6dSpUr2NXeNiKs1zbVOTJgwQeauyWjixIl2jHnz\n5sm8QoUKMn/ppZdk3qJFCzuG4tqsPvnkE7uNa1ssXbq0zA8dOiTzYsWKydz93qK2YoUQQtu2bWXu\n2gLdcfznn3/K3J2/XRvYmDFjZH4ycg1o7joQQggPP/ywzKO2VuXPn1/mronH3bds377djlGlShWZ\nDx48WOau1cfdBzszZ86UealSpew2n332mcxd6+Dvv/8u8+rVq8vctY2l5XitUaOGzF2bzJ49e2S+\nfv36SP9+7NixMne/z7S0WRUsWFDmrinVcT8Ld9+cqNHHtRy5+wrHvTd3/evVq5fMXVtbCCHkzZtX\n5tOnT5d51NYqZ8qUKTLftm2b3eaFF16QufsO6Pbl7oHc78c1UjruuhtCCHfccYfM3TXWNXG69tbT\nTjtN5u78vXPnTpkn4q7hrjF569atMr/99ttl7o77ROeHqK1VUfCXOQAAAAAAACmExRwAAAAAAIAU\nwmIOAAAAAABACmExBwAAAAAAIIWwmAMAAAAAAJBCMhw9evRo0naeIYPM77vvPpm7J2h/8803dgzX\ngDVr1qxjzC453BP5XRtOCCGceeaZMv/iiy9k3r9/f5n/+uuvx5hd8rRu3VrmrrHoq6++kvmrr74q\n8127dqVtYjEoUqSIzN3voX79+jJ3TQQh+Kf1uyesd+7cWebvvvuuHSMuhQoVkvnw4cNl7lrWorYu\nOVu2bJG5a88JwbcujRo1SuaHDx+OPrGY3H///TJ3rSrumHE/b9fo4lqU4mqKCCGERx99VOZXXHGF\nzNetWydz1/xw8OBBO3b37t0j7Ssu7jzgGmPc5ycEf+1wLVeumW/lypV2jChcS2EI/lrg8n379snc\nHZfuehnn8eqa35o3by7zunXryty1WbkGvmHDhh3H7I5P4cKFZe7aplwjjmufccfSgAED7Jzc5zoq\n93soWbKkzC+//HKZu2tcCL6xzV074vpsOa7dLYQQbr31Vpm7Fsbly5fLfM2aNTJ3rVU//vijnZPj\nzouu8cm1/7l2L9do5pqg3LnyySeflHkirmnowIEDMnc/i6htVu6+MoQQ3njjDZm71iV3HnXnE9fs\n65oWXbtkCP7clOzPlmspdK2DIYRQq1Ytmbtrh7uG58uXT+buPXfr1k3mc+bMkXm5cuVkHkIIo0eP\nlrk7/lwj19lnny1zd4/qjslETYiOa1Pr1KlTpP249jDXDOrOoXFSyzb8ZQ4AAAAAAEAKYTEHAAAA\nAAAghbCYAwAAAAAAkEJYzAEAAAAAAEghLOYAAAAAAACkkHRpswIAAAAAAMCx0WYFAAAAAACQ4ljM\nAQAAAAAASCEs5gAAAAAAAKQQFnMAAAAAAABSCIs5AAAAAAAAKYTFHAAAAAAAgBSSKT0Gvemmm2T+\n8MMPy3zTpk12X7NmzZL5V199JfPVq1fLfO/evZHHTrZWrVrJ/NFHH5V5rly5ZP7777/L/N5775X5\nJ598chyzO7EuvPBCmQ8aNMhu89prr8n8/fffj2VOTqdOnWT+4IMP2m127twp86lTp8p8yJAhMt+2\nbdsxZnfyqFSpksx79+4t85YtWyZzOgnNnTtX5tmyZZP5li1bZL5y5UqZd+/ePW0TE/LmzSvzV199\nVebbt2+X+caNG2U+Y8YMO/bs2bMTT+6/eeyxx2Req1Ytma9duzbS/tNiwoQJMv/444+TPnZcbrzx\nRpm3b98+0n7q1q0bx3TSxL2Hnj17ynzBggUy37Fjh8z79u2btoklUceOHWXerFkzu407d7hzTbKV\nLFlS5iVKlLDbLFmyROYdOnSQef/+/WWePXt2mV999dUyd9eUtMx18uTJMn/rrbfsvqLImTOnzIcO\nHWq3ueCCC2T+3XffyXzPnj0yd/ch8+bNs2Mn26mnnirzDz/8UOb79++X+Zdffilzdx3YsGHDcczu\n37njzN3ju+Mva9asMl+/fr3M3X1LCCF8/vnnMp82bZrdRqlZs6bM3Xm6bNmyMl+0aJEdw81p5syZ\nMnff805G7jg+++yzZd6wYUOZu+9Id9xxh8wTneMGDx4s8+LFi9ttlM2bN8vcnTfmzJkjc/d9PoQQ\n/v7770hzctz5tWnTpjJ3x3GVKlXsGGeddZbM3fnBXWsU/jIHAAAAAAAghbCYAwAAAAAAkEJYzAEA\nAAAAAEghLOYAAAAAAACkEBZzAAAAAAAAUkiGo0ePHk3azjNkkLkb0j0p+8CBA3aMqlWryvy8886T\nee7cuWXunlA/YsQIO3ayFShQQOannKLX4FxrlWsF6devn8zLlClzHLNLDvd0+iuvvFLmidqb3nvv\nPZm7xpC4uKecZ8mSxW5Trlw5md99990yd80Ml1566TFmlzzLly+X+aFDh2T+zTffyNy95127dqVt\nYicRd+5z58q0cOcH98T8jBkzyrx58+Yyb9y4sR3bPfk/aoNc+fLlZX7++efLfPfu3TLPkyePHeNf\n//qXzN3x98orr9h9pZf69evL/Pnnn5e5axpy18tVq1albWIRuPaHsWPHyrx69eoyd60WBQsWlPlP\nP/10HLP7Z4oUKSJz18JZtGhRmY8aNcqO8frrr8s8vZqGpk+fHnkb9/l1nznXbtO6dWuZV65cWeau\naSpRs45rMFy8eLHM3X1cVMOGDZO5e88hhHDDDTfI/Nxzz5X5ihUrZO6aj05G99xzj8zr1Kkj82rV\nqsncNURddNFFkefkrr2uxfTnn3+WeaFChWTu7o/d7zkE/72gc+fOMh8zZozdVxT58+eX+cUXX2y3\nqVixoszd79rd+3ft2lXm7h41Ldzn3bXrunYqdz2bOHGizJ977jmZz58/X+auVToE/zvaunWrUe/+\npgAAIABJREFU3UYpVaqUzF1b6eWXXy7zRNe/du3aRZqTuz921wLXKOWaulyTbaIx3H3Zr7/+KnP1\nPYK/zAEAAAAAAEghLOYAAAAAAACkEBZzAAAAAAAAUgiLOQAAAAAAACmExRwAAAAAAIAUki5tVu6p\n265FqUmTJnaMvXv3Rp9YisuRI4fMXYOJa0q6/fbbZb5gwYK0TSwduKeAhxDCpEmTZO7aptJT1NaT\noUOHytw19ETlGo5CCOGzzz6TuWsYGTRoUCxzSk+uocA1ZPTp00fmrumlZ8+eaZtYDAoXLixz15rg\n/n0Ivo1g7dq1kecVRe/evWU+ZMgQu41rynGNA+64T7ZE57iXX35Z5s2aNZO5a9TbuHFj9IlFdOGF\nF8rcnadr1qwpc9c8+T+ZazwJIYTx48fL3DV3xuWxxx6TeaZMmWT+wAMPJHM6IYQQsmbNKnPXquL+\nfc6cOe0Y9913n8xdq9jIkSPtvqJw19dzzjnHbrN06VKZu7Ya9+9do+yzzz4r82Sf7+PUtm1bmbt7\nKfezC+Hka9x07y0E367kruHp1Y4Xgv+Zu3vLm2++WebuPjtRA5Fy5pln2tdc+5H7+bnPtbsPSdTu\nnF7q1q0r8759+8rctQvOmDFD5u5aE0Lyv6+6xuxcuXLJPNG9fOnSpWXumrxdoxltVgAAAAAAACmO\nxRwAAAAAAIAUwmIOAAAAAABACmExBwAAAAAAIIWwmAMAAAAAAJBCdO1AknXv3l3mN910k8zd075D\nCOHIkSMyf+ONN2T+7rvvyjzq08zTk2unat26tcybN28u8/Rsrbrgggtk3rRpU5m7p6Wfcopfj2zQ\noEH0iUVQpUoVmVerVk3ml1xyid3X2WefLfMOHTrI3LVcxcW13oQQQvbs2WU+cOBAmf/yyy8yf/vt\nt6NPLMncZ+j666+Xed68eWV+MrZWde7cWeaPPPKIzF2DhDu+Q0i/FpPPP/9c5jfeeKPd5owzzpB5\nv379ZO4aMtq3b3+M2f0z1157rX3NtT/s2LEjWdNJKFGzzg8//CBz187hGuHce3atWK4RIj2VL19e\n5pdeeqnMb7nlFrsvd82MS69evWT+4IMPynzmzJkyX7x4sR2jTZs2Ml+0aNExZvfvXANfo0aNZO6u\nTbNnz7ZjfPPNNzJ39yhXXHGFzB966CGZ//rrrzJ3jX3uvB6Cv3detmyZzN1xeeWVV8r8q6++krmb\n69ixY2V+IrhWXNfqtHLlymROJ6Fzzz1X5q5RMVG7kuOujXG1Vp166qkyd5/Fxo0b2325hi3XjOwa\n5+L6npeWRqkSJUrIfN++fbGNkWzu+5b7jumaoNatWyfzV199VeYn4ruq+16TllZXxzU6xnGPwl/m\nAAAAAAAApBAWcwAAAAAAAFIIizkAAAAAAAAphMUcAAAAAACAFMJiDgAAAAAAQArJcPTo0aNJ23mG\nDMna9X8666yzZH7bbbfJ3DWD5MyZU+YdO3aU+ddff33syZ1guXPnlvnLL78s82HDhsl8/vz5sc3J\ncc1OFSpUkHnlypVlvn//fjvGXXfdFX1iEbgnk7vWBNd+FYJvrXJPdz8ZZc6cWea7du2SeaLGrFTn\nWp1cE93u3buTOJv/w7VTFSxYUOYPP/ywzN15JgTf+pVe7UqJuGaGZ555RuaunaNLly6xzUlJ1BD1\nwQcfyLxFixYyX7VqVSxzchLN1TW0LF++XObueHUtV67Z0rVkngjuGHOfB3fNGjdunB3j0KFD0ScW\ngTtnuSYo187iPj8hhPDjjz/KvFSpUseY3cmvRo0aMu/atavMXWteWrh2qt9++03mW7dujbT/119/\nXeaunbN27dqR9h8nd51r1qyZzF2roWtOTLSvqNy9VL169WTerl07mbt70RBCePHFF2Xu2usS3Wsr\nroHP/Vzdew4hhIwZM8r822+/lblrcXP308k+h4YQwlVXXSVz12pXtGhRmbvvESdjI7NrfHLv+fbb\nb5d5ojbHN998M/rEYuDarxK1vbrjzzVm3XPPPTJXyzb8ZQ4AAAAAAEAKYTEHAAAAAAAghbCYAwAA\nAAAAkEJYzAEAAAAAAEghLOYAAAAAAACkkHRps3JNRq7VYu/evbHNyXGNDQsXLpR5pkyZZO7aK/7+\n+++0TSwGZcuWlXnr1q1lft9999l9XXfddTL/8MMPo08sBh999JF9zbVC7NmzJ1nTSeiKK66wr82a\nNUvmnTp1knnUlivXFHD48GGZu2aYRB599FGZu89K3759I49xssmaNavMXaPLiWj4c9y5yTX2jRgx\nItK/DyGEkSNHRp9YErl2rRBCePzxx2XuGjJc49yJaMJwLr74Ypk//fTTMo+rTebKK6+UuWuLDCGE\nu+++O5axu3XrJvNrrrlG5g0aNIhl3Di5hiPXMOnaHFOJO/+EEMKGDRtk7hrQDhw4EMucTgTXUOaa\nWKNeF10TawghfP/99zJ39+Dbt2+XuWsic+fKdevWybxx48YyPxHcPb5rV3JNUDfeeKMdw7XSpJdC\nhQrZ1yZMmCDzuXPnyvxkvF+79NJLZf7VV1/J/P3335d506ZNZe5a2VwbXJxcq2aePHlkfuuttyZz\nOiGEEOrXry/zOnXqyLx3794yd9eCNWvWyDxRg5x73+5cs3nzZpm7YyZOgwYNknndunVl7s7TtFkB\nAAAAAACkOBZzAAAAAAAAUgiLOQAAAAAAACmExRwAAAAAAIAUwmIOAAAAAABACtE1M0m2ceNGmb/x\nxhsyf/DBB+2+Vq9eLXPXxnPVVVfJ/Pnnn5e5awVxHnjgAZn//PPPdpslS5bI/JdffpF5xowZZe6e\nfO3ew7PPPmvn5LinvrtmmIEDB8r8448/lrlrmnKtLWeccYbME+3LcU1dZcqUkfmkSZNk7p427/YT\ngn86vvtdR/X222/LPOpT/EMI4bXXXpO5e9/uuEwlBQoUkPn06dNlnqjxIir3VP4sWbLI3DXl1KtX\nT+au8cI1EcTZWOXegzvuCxcuLHPXvOVaxUII4eqrr5b52rVr7TZR3HbbbTIfO3aszF3jYaJzXI8e\nPWQe9boVVf/+/WXevXv32MZwjVmuzcr9+7RwjW2fffaZzN212tm6davMT8S58pxzzpG5ayxyzVHu\ns1WkSBGZDx8+3M7pnXfeiTS24xqLXEuYa+jZuXOnHaNChQoyb9mypczXr18vc3evGFWiY6Z48eIy\n79Wrl8zdPdMNN9wg8/Lly8v8zjvvtHNyXDNkyZIlZe6aztz5wV0XXSOX+z27+/W0cO/5oYcekvn4\n8eNl7j4nDRs2tGOfd955Mnc/D8cdf+67mWtxS9TY515z7YzufOIaV53Ro0fLfMiQIXabTz/9VObu\nfbufnzufjBkzxo6dbNWqVZP57bffLnPXjufO0+7z8NJLLx3H7P6du7d013D3fe6tt96S+apVq2Tu\n3lsIvmH5k08+sdscL/4yBwAAAAAAIIWwmAMAAAAAAJBCWMwBAAAAAABIISzmAAAAAAAApBAWcwAA\nAAAAAFIIizkAAAAAAAApJMPRo0ePJm3nGTJE+veu/nfAgAF2m5tuuknmGzZskPn3338v85dfflnm\nrsbMcdVqbdu2tdu4StWyZcvK3FWIuvpzV2U3YcIEO6eo8uXLJ/NHHnlE5q6211W2urrEu+66y87J\nVYI6rg7S1fG5CtYff/xR5okqj1988cXI20Qxd+5cmRcrVkzmR44csftytb2upjvZXM1q7ty57TYl\nSpSQ+ZYtW2TuqrKnTp0q848++siOHZX7DFWtWlXm7jzw22+/yXzUqFEy37x587En9w+5evChQ4fK\n3NWdumvE/Pnz0zaxGLjq3FatWsn8lFP0f1tJdBy7CvJ169YdY3b/jKsmT1QP7qqsc+bMKXNX/ek+\n73/88YcdO6ps2bLJ3B1nZcqUkfmff/4Zadx+/frJfPHixZH2k0inTp1k7ipVXe7Oie49u/NMCCGM\nGzfOvhbFZZddJvPBgwfLfP/+/TJ314EQ/PV94sSJMo9aWx8nVynevXv3SPuZN2+ezF1184oVKyLt\nPwR/L9+1a1eZHzx4UOauOvzZZ5+VebLPlYlkzpxZ5u49u9p1Z+XKlfY1d/8ftfra3Q/26dNH5j/8\n8IPMp02bZseYOXOmzJcvXy7zv//+2+4rCnd/nOh7h6u0L1q0qMzdd9KoVdmunj5O7runOwZatGgh\nc3d+GD58uMzdOTctcuXKJfMmTZrIvHHjxjJ392W7du2yY7v1BLf+4K5PatmGv8wBAAAAAABIISzm\nAAAAAAAApBAWcwAAAAAAAFIIizkAAAAAAAAphMUcAAAAAACAFHJStVkBAAAAAADg/6LNCgAAAAAA\nIMWxmAMAAAAAAJBCWMwBAAAAAABIISzmAAAAAAAApBAWcwAAAAAAAFJIpvSewD91zjnnyLx79+4y\nL1iwoMwzZswo8zx58si8bt26Mj/77LNl3rdvX5mHEEKuXLkizWnTpk0yv/POO+0YcbnuuutkXrRo\nUZm7n0eBAgVkXrx4cZnv2bNH5pky+UP4rrvukvmKFSvsNlGcdtppMh8yZEik+YQQwrhx4yKN7f79\ntGnTIu3nRMiaNavM+/XrJ/NixYrJvGXLlnFNyXKf9wMHDsg8W7ZsMj/vvPNk3rp1a5l36tTpOGaX\nHFWrVpW5O2dVrFjR7qtQoUIyT69mw0TnRHd+rVy5sswPHz4sc/c7PRm562L//v1l7q5NaeE+KyVK\nlJD5ZZddFunf58+fX+a7du2SuftZpKfs2bPLvGvXrnaba6+9VuaXX355LHNyqlevLvN69erJ/OKL\nL7b7csfZjz/+KPMuXbocY3bHp3nz5jLv0KFD5H1Fvbd0n7m33npL5hMmTJD5VVddZee0e/dumb/3\n3nsyf+WVV2Qe1z1TIu4+oV27djKvWbOmzN09p7tfmzNnjswfeeQRmbt70fSUN29emV944YV2m9q1\na8s8S5YsMu/du3f0iSWZe39PPvmkzN0x5o4lp3379va1WrVqydydB9xx6f795MmTZT5y5Eg7p6ge\ne+wxmbtz3KJFi2R+5MgRmefOnVvm7vvfqlWrZB5CCIMHD7avReG+wz7wwAMyz5kzp8zdPXAIIWzZ\nskXm7j0sWLDA7uu/4y9zAAAAAAAAUgiLOQAAAAAAACmExRwAAAAAAIAUwmIOAAAAAABACmExBwAA\nAAAAIIWkRJtVoieH9+nTR+bPPPOMzB966CGZ58uXT+bffvvtMWb379xTqe+//367zW+//RZpjNGj\nR0f69+4J7i1atJD5mDFj7L4+/PDDSGMn25VXXmlf++KLL2TuGouc888/X+ZLly6VuTvG3BPcQwhh\nx44dkea0fv16mRcpUiTSfuLkGspmzZolc/d7uO+++2KZj2vLCiGEn376KdLY7ue6evVqmbun77um\nsxPBNTu5NpxrrrlG5gcPHoxtTlFdeumlMl+3bp3ME7UxFS5cWOajRo2S+YwZM44xuxNv7ty5Mj/z\nzDNl/v3338s8ruYj1w4UQggTJ06UuTtPu/yXX36RuWsBWrx4sZ1Tsrl2Ktca5K7J7mcXgm/biOqS\nSy6R+QsvvCBz15w4adIkmQ8aNMiO7ZquXn75ZbtNFO4eKHPmzDK/8cYb7b6iXqvPOOMMmbuWNXcc\nuzm5FpYQ/HXr0Ucflfny5ctlfiLaCF977TWZu/vR559/PtL+3X3csmXLIs3HNaydCDVq1JC5+yy6\n4z4E/7tOdP+fXho0aCBzdz/g7kXnzZsXaVz33bNMmTJ2mwcffFDm7h7FcW1WrlXTHQMbNmyQeaJr\ntbtmvvvuuzIvXbq03Zfyww8/yPzZZ5+V+d69eyPtPy1+/fVXmd97770yT8ucXnrpJZm7c1m1atWO\ne9/8ZQ4AAAAAAEAKYTEHAAAAAAAghbCYAwAAAAAAkEJYzAEAAAAAAEghLOYAAAAAAACkkJRos3r9\n9dftayNHjoy0r+rVq8t8woQJMr/55psj7f/WW2+N9O8Tueeee2T+8ccfR9qPa6Po379/5DklW7Fi\nxWRer149mbsmnhD8U+Wjck/9z5Qp+R+fhx9+WObjx49P+thOoUKFZO5a2T755BOZV6lSReZ79uyJ\nNJ9XX31V5olaR1yDhWt0SSXu57F582aZ161bN5nTSRPXRjF16lSZu2Yq19qSnrJkySJz19xz9913\n231df/31Mt+6dWv0iUVw3XXXyTzRtbpChQoyX7RoUSxzOhFc21StWrVk7pph3Ge0Q4cOMj8RbR6u\nwcSdpx3XpHbHHXfYbdq0aSPzpk2bRhrb2b9/v8zjvI66ZizXANqoUaNYxt24caN9rWzZsjKvVKmS\nzF2D04mQqEFMyZEjh8zdseQ+W+76l56tVU758uVl7lqrHnvsMbsvd2+ZXlzbZgghtG3bVubufXfr\n1k3mURspXWvVRRddZLdp0qSJzF1TU1R//fWXzN09kGuzctegEPx1zl2rU+kaHlXUa2/r1q3ta+4+\nIeo6g8Jf5gAAAAAAAKQQFnMAAAAAAABSCIs5AAAAAAAAKYTFHAAAAAAAgBTCYg4AAAAAAEAKSYk2\nK/f07hBCuPjii2XunhzunkxdsWJFmW/ZsuUYszs+p512mn3tmWeekXmXLl1k3q5dO5lfeOGFMndN\nDqtXr7ZziqpBgwYyd00s7udRs2ZNma9atUrml156qZ3T9u3b7WvpwTVIhBDCU089JfOiRYvKvESJ\nErHMKS2KFy8uc3ecueabQ4cOxTYnxbXBheDbCGbNmiXzsWPHyjxRk0iyNW/eXOZ58uSR+e23357M\n6cTq+++/l/mgQYNk7o69k9G9994rc9dmlexmqrRwLVrTp0+32/z888/Jmk7s3PXp7bfflrlrcNq1\na1dsc0o2dz52LUAur1y5ssyfeOIJO7a7Nib7GpEWVatWlfngwYNl3q9fP5nPmTMnlvkMHTrUvta5\nc2eZu4abggULRspPxPXPNdC6Np4RI0bIvFq1ajJPpWuHe2+u7cnlIYQwYMAAmR88eDD6xGJwxRVX\n2Nfc97wePXrIPGprlZPoHjKqhg0byrxPnz4yP3z4sMyPHDki86jtut27d7evuQbk7Nmzy9zdu7hj\nLNF3+vSSK1cumbtGRXc/7a4PIfhr44IFC44xu2PjL3MAAAAAAABSCIs5AAAAAAAAKYTFHAAAAAAA\ngBTCYg4AAAAAAEAKYTEHAAAAAAAghWQ4evTo0aTtPEOGZO36P7kn3bun77undJctW1bmcbVZDRs2\nzL62ZMkSmU+ZMkXmffv2lXmnTp1kXqBAAZnv2LHDzim9uBaR2267TebuZxFCCOedd57M43qSupvr\nsmXLZL5p0ya7ryFDhsjcvb/nnntO5uPGjbNjxMU1t7jjeOXKlTJ3jTh79uxJ28T+myxZstjX3Of9\nggsukHmrVq1kPm3aNJkn+rzHxTU8DB8+XOabN2+W+ZgxY2Q+efLktE0sAnduci0prsnBtVw99NBD\naZtYErlGCNeY5toU0tMpp+j/DtS+fXu7TceOHWW+dOlSmSdqYkkvn376qczLlSsn88svv1zmK1as\niG1O6cXdY9WpU0fmidqsXDPWxx9/HH1iMXD3iSGEULhwYZm7Js6SJUvKPK57yzjVq1dP5q+88orM\nR44cKfP+/fvHNifX3HnLLbfIvHbt2jLfuXOnzBcvXizzl156Sea//fabzBNxTTlx3YO7c+Vrr71m\nt3G/67iaoOI0depUmY8aNUrmrnUwPZ1++ukyd99H3L1Ovnz5ZL527VqZZ82a9diTO07ufs01dLp7\n/HPPPTe2OcWlcePGMnfXdnff4lqlQwihZ8+eMr/ssstkvnz5cpmrZRv+MgcAAAAAACCFsJgDAAAA\nAACQQljMAQAAAAAASCEs5gAAAAAAAKSQ41rM2blzZ2jevHkoU6ZMuOCCC8LXX38dtm/fHurWrRtK\nlSoV6tWrZx8uBgAAAAAAgPgc12JOjx49QqNGjcKyZcvCkiVLwvnnnx8GDx4c6tatG1auXBnq1KkT\nBg8enOy5AgAAAAAA/K93zGryXbt2hQoVKoTVq1f/W37++eeHOXPmhPz584dNmzaFWrVq/T81Wq6a\n/MEHH5S5q/386quvEk0xElfbu2HDBpknqtNML/Xr15d5t27dZH7ttdcmczrp6uDBg/Y1VyF6MlaC\nOh999JHMXeXk+++/H2n/p556qswPHToUaT8hhJAjRw6Z//jjjzJ3c+3Ro0fksdPLPffcI3NXmblt\n27ZkTidNnn32WZm7OlpXyZgWdevWlbmrR3XVmB988IHMq1atmraJxSBv3rwy37p1q8xdNXnr1q1j\nm1N6cnXmrn5zwIABMh8zZkxsc4pLjRo1ZO7qjcuWLZvM6ZyUunfvbl+79dZbZV6lSpVkTSd2L7/8\nsszfeecdmU+fPj2WcWvVqmVfczXxTz75pMx3794t82zZssl84sSJMnfXOfd7PhHctaNhw4Yyf/TR\nR2U+YsQIO8aQIUNk/sYbb8jc3X+5qvG///5b5r169ZJ5oor4Sy65ROYLFiyw26SXw4cPy9zVbru6\n7/8J3Gf69ddfl3mhQoUij5E9e3aZ7927N9J+3PKCq1dPpe9mjvtMhxDs/8H08MMPy3zo0KEyT1M1\n+Zo1a8JZZ50V2rZtGypWrBg6dOgQ9u3bFzZv3hzy588fQgghf/78YfPmzcfaFQAAAAAAAP6hYy7m\nHD58OCxcuDB07do1LFy4MGTLlu3/+V+qMmTIYP8KBwAAAAAAAPE55mJO4cKFQ+HChUPlypVDCCE0\nb948LFy4MBQoUCBs2rQphBDCxo0b7Z9NAQAAAAAAID7HXMwpUKBAKFKkSFi5cmUI4f8816Zs2bLh\n2muv/c//d33MmDGhSZMmyZ0pAAAAAAAAQqbj+UfPPfdcaNWqVfjrr79CiRIlwqhRo8KRI0fCDTfc\nEEaOHBmKFSsW3n777WTPFQAAAAAA4H+941rMKV++fPj222//n9y1Tx3LY489JnP3pOy0tFmdfvrp\nMr/uuutk3qpVq8hjRHHZZZfZ17744guZX3jhhTJ3T8xv06ZN9InFxM3VNd9kzJhR5tWqVZO5e9q3\na4oIIfqT0V2DgGsmiqp8+fL2tWeeeUbm//G/Mv53UVur3JPuy5QpI3PXOBeC/5zWrFlT5q4JY+3a\ntXaMKFwLkGsNCiHYxeeff/5Z5u7p/uedd94xZvfPub96dA1OUWXKpC8Drv0jzjarG264QeYzZ86U\n+VtvvSVzd3ynJ9dE5xq83L+/8sor7RizZs2KPrEkcp/1EPy1132GFi9eHMuc0uK0006TuWtJyZw5\ns8yPHDkS25ziUrFiRZkvWrRI5q5Bp2jRojJv3769zDt37mzn5Np4olL3qSH4+wR3L5WIu8655rxE\nLV5xmD17tn3NfbbGjx8vc9d8M3r0aJm7c9b9999v5+S4Nsx7771X5gMHDpS5+8zlyZNH5u6c5faz\nZMkSmSfSs2dPmX/zzTeRxnA/o9y5c8u8Xbt2dk5xtVa59zZs2LBI+7ntttvsa999953Mk91addNN\nN8n8zTffjLwv953HfUavv/56mbvza6JWO8V9ZwvBH5fuM/T444/LfN68eTI/Ea1Vrq1t4cKFMnfX\nOSdnzpwyf/755+027nPt2u6iOOb/ZgUAAAAAAICTB4s5AAAAAAAAKYTFHAAAAAAAgBTCYg4AAAAA\nAEAKYTEHAAAAAAAghRxXm1XcpkyZIvOWLVvKPFGbR9asWWW+f/9+md95550ynz9/vh0jikqVKsn8\ngQcesNscOnRI5rt375a5a8Zy7zlOrtXHtcysWLEi0n5+++03mffu3VvmroEjLVybVevWrWW+Y8cO\nmefKlUvmy5Yts2N36NBB5qtWrbLbRNGjRw+ZP/HEEzJ/8skn7b7+/PNPmbv3V69ePZnH1abw6quv\nyrxs2bJ2G/c7veiii2S+bds2mbtmPvfv08I9NX/ixIkyz58/v8z37dsn888//1zmaW0rjOKnn36K\nlH/99dcyHzFiRGxzikufPn1kvmbNGpnnzZtX5o0aNYptTnE599xzZe5aR0LwrTuu5S8trTFxeeWV\nV2TeuHFjmf/6668y79q1a2xzios7P7jfT758+WS+YcMGmU+fPl3mpUuXtnPauXOnfS2KGjVqyPyp\np56S+dy5c2XuWiRDCGHdunUyr1ChgszTs9HMHccudz+/Zs2aybx69eoynzZt2nHM7t+dffbZMnet\nO67x0DXRufsTd01xv8+0HKubN2+WuWtw6tevn8xdy9Ajjzwi8xPxneCss86SubuGu+Y/dz4JIYSG\nDRtGn1gMXLtbohZdd3/suGPDnZvceTTqeSZRK+nIkSNlvnr1apl/+eWXMnf3/ieC+47pfnfuc33m\nmWfK3B3Hic59nTp1knkcn1P+MgcAAAAAACCFsJgDAAAAAACQQljMAQAAAAAASCEs5gAAAAAAAKQQ\nFnMAAAAAAABSSIajR48eTdrOM2RI1q4BAAAAAAD+x1PLNvxlDgAAAAAAQAphMQcAAAAAACCFsJgD\nAAAAAACQQljMAQAAAAAASCEs5gAAAAAAAKSQTOk9gWQpWbKkzIcMGSLzhg0bynzBggUyv/zyy9M2\nsRhcd911Mh86dKjMs2XLJvN33nlH5nfeeWfaJpZEp556qsxvv/12u83FF18s806dOsn8mmuukfm0\nadNkfujQIZnnzp1b5o0aNZJ5CCHUrFlT5jly5JD5jTfeaPeVbFdeeaXMmzdvLvPMmTPLPG/evDL/\n9ddfZR7ncVm3bl2ZlylTRua1a9eWuTsun3/+eZl/+umnMnfHUpxq1aol88qVK8v8kksukfmsWbPs\nGC+++GLkeUXx9NNPy7xnz54yX7t2rd3X77//LvPixYvLfNWqVTK/4oor7BjJVrVqVZk/8cQTMnfv\n4ZFHHpG5+yyeCKecov9bU7NmzWTep08fmW/evFnmDRo0SNvEYuDOPzfffLPMv/jiC7uzRkPHAAAg\nAElEQVSvHTt2yNxd35Mtf/78Mh8/frzdxp2bBg0aJPOHHnpI5hMmTJB5kSJFZL5z506Zr169WuYh\nhPD111/L/LvvvpP58uXL7b7iULBgQfuau69w90Znn312pNzdG7n77JNRpUqVZN6yZUuZu/uZEELo\n0aOHzG+44QaZv/HGGzJ35+mzzjpL5u46V758eZmHEELGjBll7q7hXbp0sftKNvcd5p577pF5586d\nZT579myZ33TTTWmal+LuCadMmSLzChUqyHzXrl0ynzRpksznzJkT6d/HqXDhwjK/5ZZbZO7up93x\n7b7bhhDCuHHjjjG7E8t91kMIoU6dOjLPkiWLzNu0aXPc4/KXOQAAAAAAACmExRwAAAAAAIAUwmIO\nAAAAAABACmExBwAAAAAAIIWwmAMAAAAAAJBCMhw9evRo0naeIUOydv2f+vXrJ3PXzuGefu5aO5Yt\nWybzpUuXHntyJ4mrr75a5gMGDJC5azqIU7FixWTuft7uad+LFy+2Y7jGJ9cu8dprr8m8bdu2MndP\nm3dPs9+zZ4/MQ/ANIK7lwz25PirX+jZ8+HC7jXs6/quvvirzI0eOyNy9502bNsncnU/cz/uDDz6Q\neQj+eLrvvvsizSl79uwyd8exazSbOnWqzONUrlw5mbvWoAsvvFDmrs0shBDOPffcyNtEcdppp8nc\nNXOULl3a7uv666+XeYcOHWQ+f/58mbdo0cKOkWw5c+aUedmyZWXumrdco8uJeG8XXHCBzAcOHChz\n1zQ0Y8YMmX/++ecy37dv33HMLjnOOOMMmU+fPl3mI0eOtPtybV0ffvhh9IlF4D6LP//8s8y7detm\n9+Vakf76669Ic3INiVmzZpW5O1+5624IvsnUNbS447hv3752DMVds+bNm2e3cce4uzbu3r1b5lWq\nVJH5eeedJ3P3szgRXLuXu/a66+LMmTNl/tlnn9mx9+7dK/OrrrpK5hs3bpT5H3/8IfMtW7bYsRXX\nDhRCCHPnzpX59u3bZX7ZZZdFGttp1aqVzBO1Eh0+fFjmrh30rrvukvmKFSuOMbt/zt0Hu/O0+67q\n7l3cZ+79998/9uSSxN1LufNAvXr1ZO5aserXr2/HTnT+i4O7P3af6US/B9fy59pHDxw4IHO1bMNf\n5gAAAAAAAKQQFnMAAAAAAABSCIs5AAAAAAAAKYTFHAAAAAAAgBTCYg4AAAAAAEAKyZTeE/inmjVr\nJvMXXnhB5s8995zMo7YmpKds2bLJ/NFHH5W5ayK45pprYpuTU6hQIZl36tRJ5lWrVpW5axvbsWNH\n2iYmtGvXTuauAe3MM8+UuWtneemll+zYlStXlvl3331nt4lDw4YNZe6euh5CfE1aw4YNk3nXrl0j\n7adly5Yyr1atmt3Gtf24dgn3lP2hQ4fKfMGCBTJ37QsnwpIlSyL9e9fgde2119pt3njjjUhjROXO\n088++6zME7UxPfzwwzIvUaKEzP/8889jzC45SpUqZV9z5/bcuXPL3DW6uCaoOLmWxPHjx8vcfX5d\n487JyLUlueP1lltukfmqVatim1NcXFukaxv76KOPkjmdEEIIW7dujfTv3X2Fuz6E4Juu3Pl14sSJ\nkebkuGaTSy65JJb9hxBCrly5ZO7uB9xxfCK4Bp3BgwfLvE2bNjJP1BQXl7iu+88884zMGzRoIPM1\na9bYfbnmTtdWGlWePHlkftttt8n8RLQfx8U1qIYQQvv27WXufkeuFdA1Jp+MTcpRvwu5e5pRo0bJ\nPNmNVYm488bdd98t80TXjrVr18YxJYm/zAEAAAAAAEghLOYAAAAAAACkEBZzAAAAAAAAUgiLOQAA\nAAAAACmExRwAAAAAAIAUwmIOAAAAAABACkn5anJXrfzFF1/I3NVxlytXTua7du1K28RiUKNGDZnP\nnTtX5q6+7fLLL49tTlG5KjZXi7h48eIkziZttm/fLnNX0Tx69GiZL1y40I7hagv3798v859++snu\nK4rnnnsulv2E4OvmXVVj0aJFZX7zzTdHGnfcuHGR8hB8feC//vUvmdepU0fm2bNnl/nw4cMj/fsd\nO3bIPE7uXNmtWzeZuxrr8uXL2zGi1p/HJWvWrDI/ePCg3ea6666T+ffffy9zVzefbK6uMwRfA+0q\nmgsXLizzWbNmRZ+Y0LhxY/vaBx98IPM//vgjlrHTkzt/T506VeZvvPGGzG+66SaZFylSxI59xx13\nyPzQoUN2G8Wdj1199+effy7zMWPGyHzatGl27IEDB8p8zpw5dptk+vjjj+1rrpq8atWqMnfXjkWL\nFkWeV1zcXF9++WWZu5rpV199NbY5ubpnV2lfv359mb/44osyP+WU1P9v1/3795f5a6+9JvPnn3/e\n7uuss86KZU7O448/LvPOnTsnddwT4ciRI/a1Pn36yHzw4MEyd/eiAwYMkLk7V0Y936eFqxRv2rSp\nzH///XeZr1y5UubunjNv3rx2Tu5eJy6ffvqpzIcMGSJzd24NgWpyAAAAAAAA/P9YzAEAAAAAAEgh\nLOYAAAAAAACkEBZzAAAAAAAAUgiLOQAAAAAAACkkw9GjR48mbecZMiRr18eUMWNGmc+ePVvmrlGj\nWbNmcU0psmzZssncPYXetYU0b95c5qtWrUrbxCIoUaKEzN99912ZuyeBHzhwILY5ReVaFlyDgGvg\n2LZtmx2jQIECMu/atWuksYcNG2bHSLa+ffvKfOnSpTLv2bOnzF0bmMvTInPmzDKvXLmyzL/++muZ\nuxY814by/vvvy7xLly4yj9Npp50W6d+ff/75MndP9w/BNzNs2bIl0thxce1hIfhz/vz582Vet27d\nWOaUni655BKZP/jggzJv0qRJpP27xrQQQvjmm29k3qNHD5mPHz9e5itWrIg0pxPBXWPd/UbUBo4H\nHnjAvuauN67Rs2XLljJ358SxY8ceY3b/LkeOHDLv3bu33ca9v0yZUqdw1Z37XLuXa7O6/fbbY5uT\na3xy94ruWHLnylatWsn8iiuukLk7tybifq5ZsmSReb58+WTujrHNmzfL3LVqnojWoLicccYZ9jXX\nipuorTKKSpUqyTxRy6hTpkyZfzqdWLnvZons27dP5hdccIHMXbPqjz/+KHP3XeFkdPHFF8vcNYnW\nq1fP7mvGjBmxzCmq1q1byzxRg1yiz2MUatmGv8wBAAAAAABIISzmAAAAAAAApBAWcwAAAAAAAFII\nizkAAAAAAAAphMUcAAAAAACAFJIuVQFt27aVuXtKt2vBSOTIkSMyf/rpp2X+0ksvRR4j2dzTz13+\n559/Rvr3aeF+Tg8//LDMf/nlF5m/8MILMr/qqqtkPnny5OOY3T9zyil6bbNTp04yv/XWW5M5nRBC\nCOvXr5f5448/LvP0bLMaMGCAzE8//XSZ33///TL/7bffIo2bN29emW/fvt1uc/DgQZnPmzcv0tgL\nFiyQuWtsqF69eqT9hxDCpZdeKnPXsOX89ddfkf69azDZu3ev3SbZrVWupdA1qbnzUgj+eHV5VK79\n6vLLL5f54MGDZb5///5Y5hNCCDt27JB5/vz5I+2nRYsWMp84cWLkOblGuCFDhkTeV1zcteDvv/+W\n+TvvvJPM6YSSJUva11xDnuNafdq0aRNpP86ePXtkfvbZZ9ttXnnllVjGjkvWrFnta64hyrWHLVmy\nROaJmvbiMm3aNJm7a3JUrgnKNbpUqFAh8hgrV66UedRWqZkzZ8rcNct17NhR5iNGjIg0bnpKdP1z\nbYFxcfdGrpkq0bUjZ86cMt+1a1f0iUXgmtFcw1oIIaxevVrm7nvYTz/9JPP27dvLfNKkSXbsKNx5\nLATftOc+71EVLFhQ5u5e52RssNy5c6fMXZtjsvGXOQAAAAAAACmExRwAAAAAAIAUwmIOAAAAAABA\nCmExBwAAAAAAIIWwmAMAAAAAAJBC0qXNqlWrVjJ3LTaJ2jyWLl0qc/f08+HDh8s8rjYr19Q1Y8YM\nu82GDRsijeGaW9atWyfzjRs3Rtp/Iq6F6ttvv5V5vXr1ZO5afcaOHZu2icXANZUUKlRI5r1795b5\nwoULZZ4lSxY7dtOmTWXeqFEjmbvjLC6uNSiEEGrXri3zGjVqyDxXrlwy79Kli8zdk/Sdhx56SOau\nNSEEf5y5Nqu1a9fKvGrVqjK/5ZZbZP7iiy/aOTnbtm2T+aBBg2Tev39/mbu2u2LFisnctV24339a\nuOPszjvvlPmTTz4pc3f+TtRKE7XdK6pevXrJ3LVIuOM1UcuQa4WoU6eOzF1Dy2OPPWbHUNx5KdHP\n1DXouCatOFsYo7r77rtl7lp6Dhw4EGn/7lrgfg+unTOEEJYvXx5p7C+//FLmrk3m4osvlrm7L3Nt\nWdWqVbNzKl++vH0tmU499VSZf/bZZ3Yb1yZz8803y7xWrVoyj9pseP3118vctYeFEP0c5/69u7a7\n+5CorZAhhPDUU0/J3P2OBg4cKHPXEtauXTuZu1aajz76SOYnI9fAeM0119htEjUyReFal0aOHCnz\nEiVKyNwdYyEkv7XKce/BHZMh+HsRd51zzYmu7dW1Pkfl7r9D8K1z7733nszXrFkjc3d/4hqCe/bs\nKfNff/1V5ok0aNBA5p988kmk/RQoUEDmrVu3lnlazn1x4C9zAAAAAAAAUgiLOQAAAAAAACmExRwA\nAAAAAIAUwmIOAAAAAABACmExBwAAAAAAIIVkOHr06NGk7TxDBpm7BpMhQ4bIvGHDhnYMty/X3PLu\nu+/K/PHHH7djROFabO644w67jWv5yJYtm8xnzZoVaYxDhw7ZsePiGhtGjRol819++UXmrv3KNU2d\nCK4R58EHH5T5tddeK/P169fbMb766iuZjxkzRuauucxxT4kvXry4zBO1B7inwc+fPz/SnJItUTtC\nkyZNZO4ahc4991yZb968WeYTJkyQuWtHSMtn9JJLLpH5vffeK/OVK1fKPEeOHDLv16+fzONslnj6\n6adlnjlzZpnfd999Mt+7d29sc4qLa3IYPHiwzMuVKydz13YRQggLFiyQ+ZIlS2Q+btw4mW/dutWO\noeTJk0fmU6ZMsducdtppMm/RooXMV61aFWlOcXINho888ojM3WfCtWpu2rRJ5h9++KHMv/vuO5nH\nybVQdevWTeZbtmyR+dy5c2Xu2t1CSH6znOOadSZPnmy3ce/7m2++kfnQoUNlHrVl1LXAJmoscg06\nZ5xxhsxdE5RrpHQtj82aNZN5ouZW1wzpGvjcNdk1y7n7E9dweyIalNx9s2vqKl26tMzffvttmd91\n11127IMHDyae3HGaOnWqzN17c22liY5j1+iZbK61yn0WQ/D3/+767o4zd21393e7d++2c1IS3Ve4\nljrXrHnWWWfJ3H3mXHPitGnT7Jyicm1trn3NtS26e3P3PTzRWkJc38XVsg1/mQMAAAAAAJBCWMwB\nAAAAAABIISzmAAAAAAAApBAWcwAAAAAAAFIIizkAAAAAAAApJF3arAAAAAAAAHBstFkBAAAAAACk\nOBZzAAAAAAAAUgiLOQAAAAAAACmExRwAAAAAAIAUwmIOAAAAAABACsmUHoP+8ssvMj/ttNNkvnbt\nWruvbdu2yTxjxowyf+CBB2S+dOlSO0YUWbNmlXn79u3tNldeeWWkfRUoUEDmO3bskHnJkiVlPnbs\nWJn37dtX5iejCy+80L7WrVs3mXfu3DmWsS+++GKZu9/nww8/bPe1atUqmX/++ecynzRpUqR/75xy\nil7PrVixot3mmmuukXnlypVlXrp0aZn//vvvMm/UqJHM9+7dK/PChQvL/Mknn5R5ojllyZJF5tOm\nTZP5+++/L/PFixfLfNeuXXZOyda2bVuZ33333TJfuHChzIcPH27HWLBgQfSJJVGJEiXsa7Vq1ZJ5\nqVKlZP7NN9/I/N133408L+XUU0+V+aBBg+w2tWvXlvm+fftk/tdff8n89ddfl/no0aPt2HGpVKmS\nzKtXry5zd5/g3vOhQ4dk/uqrrx7H7I7PkCFDZH7kyBGZHzx4UObnnHOOzM8//3w79p49e2Rev359\nu016cNeaEEIYM2aMzN09zZ133hnLnJwaNWrY11q1aiXzKlWqyNydEx966CGZb968+RizO/GqVq0q\n8+bNm8vc3Zc1aNAg8tinn366zCdPnizzL774Qua//fabzPfv3y9zd964/fbbZR5CCHXq1JH5K6+8\nInP383Dv+aeffpL57NmzZT5u3DiZhxDCypUr7WvpIdH5Knfu3DIfPHiwzL///nuZu++F11577TFm\nd+K1bt1a5i+//LLMN2zYIHP3/e9EyJMnj8ynTp0qc/ce3HU0hBBatGgh8zZt2sj8jjvukPmKFStk\n7n5+8+bNk7m7FwghhK1bt9rX/in+MgcAAAAAACCFsJgDAAAAAACQQljMAQAAAAAASCEs5gAAAAAA\nAKQQFnMAAAAAAABSCIs5AAAAAAAAKSTD0aNHjyZt5xkyJGvX/8nVkj366KMyd9Xkruo5KrcfV7EZ\nQgjdu3eXuascjcrVRP7www8yPxG/t6gaNmwo8ylTpthtXPVshw4dIo2dI0cOmbuqy169esnc1VOG\nEMIFF1wgc1dP72r03DHjqkVnzJgh80Q1st9++63My5UrJ/NcuXLJfOTIkTLv2LGjzP/++2+Zv/XW\nWzJft26dzEMI4ZlnnpF5mTJlZP7II4/I/NJLL5W5q5meMGGCzG+66SaZJ+IqE8eOHSvzn3/+WeaP\nPfaYzF21YyKubjeqM888U+YffvihzJctWybzP/74w47h6ua//vprmbuq2mQbOHCgfc0dx+56c889\n98i8UaNGMr/88suPMbvj5yrFXQWr+10n+p2ml40bN8rc1ZRu2rRJ5suXL488tqubdzXGyVa0aFGZ\nu/rxEPx7eP/992X+4osvRp+YcP3118vcvYcQQpg/f77Mt2zZInN33Xf3D23btrVjR5UzZ06Z9+nT\nR+YtW7aU+cKFC2Xufj/vvPOOzP/880+Zp4Wr745zjLi44+nXX3+NtB93zW/SpInME9Uku3uOt99+\nO9KconI144nuj5s1aybzmjVrytxdt7788kuZf/bZZ3bsZDvnnHNk7qqvixUrJvNE9d3Jli1bNpm7\nn/ewYcNk7q7t27Zts2N/9dVXx5hdcrjz9P3332+3KV26dCxjq2Ub/jIHAAAAAAAghbCYAwAAAAAA\nkEJYzAEAAAAAAEghLOYAAAAAAACkEBZzAAAAAAAAUkim9J7Af5U9e3aZuyfphxDC5MmTZV6lShWZ\n7927N/rEImjcuHHkbVzLxy233CLzc889V+YFChSQ+dVXXy3ziy666Dhmd3JwzUSu9SaE+Foh3BiL\nFy+WeaKn8jvJbh5xT3x3LVelSpWy+3KNcK61yjVm9e/fX+autcrJkyePzD///HO7ze+//x4pd20H\nrm3MtVzlz5/fzsmpW7euzB9//HGZd+3aVeYLFiyQuWuKe/DBB2VeokQJmcepUqVKMnctYZ06dUrm\ndNKVa4QIIYS5c+fK3H1+XVPXDTfcEH1iQqLPXJEiRWTumvncedc1GSX72h6CbwdyY7v7kwoVKsi8\ncOHCkfaTngoWLChzd10cPny43Ve/fv1imVNUro0pTu686+7Xojr//PPta0uXLpX5HXfcIXN33nBt\nYydCz549Ze4acU5GUVursmTJInN3r+PafrZu3WrHSNSwm0zu/iRfvnyR9zVnzpxIeXpyv7u1a9fK\n3F0L0rO1yr0H9/1i9OjRMh81alRcU0q6q666SubdunWT+YgRI5I5HYu/zAEAAAAAAEghLOYAAAAA\nAACkEBZzAAAAAAAAUgiLOQAAAAAAACmExRwAAAAAAIAUclK1WeXIkSPyNpUrV5a5a044ePCgzMeM\nGSNz95TuON10000yHzlypMzHjRsnc9dU0qZNG5kfOHDgOGZ3cnBtT2effbbdxrWERbV+/XqZu8Yd\n12Tkfj/pafv27TJ/8skn7TYbNmyQ+TPPPCPzU07Ra8auiWfAgAEyf/nll2XumjnGjh0r8xBCaN++\nvcw//fRTmbs2tXnz5sl81qxZduyoXJPWmjVrZO5awq699lqZu7aV3r17y/yFF16QeQghfP/99zJ3\nDWhO2bJlZe7a1yZNmiRzd+yFEEKmTPryN3XqVJknauNJpo8++si+9sMPP8jcNUG5tin3mY7qiiuu\niLxNxowZZV6vXj2ZL1q0SOaNGjWS+cqVKyPPyXnttddkXrJkSZm7lqZ33nlH5q1bt5Z5rVq17Jzu\nuece+1oytWzZUubumEyvxqpEXLtgxYoV7TbuvsL97jZt2iRzd92K6o8//rCvubauXr16ydx9hiZM\nmCDzt99+W+ZxNu7s3r1b5meccUakf59KXJORu5d3TYGJGqLctdTdb0RtGXXc+cpdB0Lw545HH300\nljnFqXr16jL/+OOPZT5z5kyZu++eNWrUkLm7R00L11rl2tHcOf+5556LbU5RuePbtWG6dmd3n+1k\ny5bNvlaoUCGZuxbdKPjLHAAAAAAAgBTCYg4AAAAAAEAKYTEHAAAAAAAghbCYAwAAAAAAkEJYzAEA\nAAAAAEghGY4ePXo0aTvPkCFZu04z98R018Tjnig+fvz42OaUbPfdd5/M8+bNK/N7773X7qtYsWIy\nX7t2baQ55cyZU+YDBw6MtJ/ixYvb11x7T1xNC65B4Mcff5R50aJF7b527twZy5xSiWuGKVOmjMyr\nVasW29ilS5eONMY555wj89q1a8vcNZi4Bpi0mD59usznzJkjc3eO27dvX6RxCxYsGHlOrinAcS0V\nro3JNXAkkjVrVpm71jnXAOPec3pyzZCu+cY1MMTVYBIn1yLo2utcM1paNGzYUOaumW/v3r2xjLtt\n2zb72nnnnSdz11TouM/15s2bZd6iRQuZv/766zK/7LLL7NjffffdMWaXHO4aftttt9ltOnToIPPZ\ns2fL3LVcnYxci9JTTz0lc9fs1bRp09jm5L6i/PzzzzIvVapUbGOnukT3nDNmzJC5a0C7//77Y5mT\na/RJdI5z9yiukTI9ue827lrgvo+4pl73ndR9N0vL950sWbLIvHHjxjK/8MILZe6at6688srIc4rK\nzclZunRpLOOeeuqp9rURI0bI3DVuuvOuOifylzkAAAAAAAAphMUcAAAAAACAFMJiDgAAAAAAQAph\nMQcAAAAAACCFsJgDAAAAAACQQv7XtVlF9cMPP8g8ajtLIq5Vyj2FfMeOHZH275oi2rdvL/PFixfb\nfXXq1EnmL730UqQ5RW2zcq0JrlkiLdzT4NevXy9z9/u57rrrZO7aP0II4dZbb008uRRWq1YtmbsW\nIHeMjRo1KtK4iZ4qf+jQoUj7yp8/v8wHDBgQ6d+7hrWTUc2aNWXet29fu80999wj8yVLlsQypzgV\nKFBA5vPmzZN58+bNZb5o0SKZu4aozp07y9y1ebiWphB825Q7L7rPYqtWrewYJxvXSHn48GGZp+U+\nxI3hmvZcG8Vff/0VadzTTz9d5uvWrbPb5MuXL9IY7vpUvXp1mT/wwAOR9t+7d2+ZDxo0yG7jxhg8\neHCksU8Ed253jZ4dO3aUuWv9clwrm2vPPBH2798vc9cotWHDBrsv1/q1evVqmVeoUEHmzz33nB0D\n/1eDBg1k7lpGXQuVaziaNGlS2iYmuPOfaxmNi7uGR20AjdPIkSNlPm7cOJnPmjUrmdMJIYRwyin6\n70Jcy+iJaLM6Gbk2zC5dusjcXatpswIAAAAAAEhxLOYAAAAAAACkEBZzAAAAAAAAUgiLOQAAAAAA\nACmExRwAAAAAAIAUkim9J/BfXX311TJ3T8wPIfqTuqtWrSpz15owe/bsSPt3LRiffvqp3WbYsGEy\n//jjj2V+2WWXyXz06NEyf/PNN2WeqLXKyZ07t8xdE4FrYtm1a5fM//Wvf8m8fv36xzG7f8b9vC+4\n4AKZ9+nTR+buKf7nnXde2iYWg169esl8zpw5MncNaCGEULRoUZl369ZN5u7YcL/rt956y46tuCfj\n9+/f327jzhs1atSQecWKFWXuGrbuuOMOO3ZU7v0VKVJE5h988IHM3fnVNWy5Bgm3nxCit4Q1adJE\n5u68u3PnTpmff/75Mm/btq0d++KLL5a5a1NzrVWOO76LFy8u84cffljmrskvhBBKliwZaV9169a1\n+4ri6aeflnmi5q2oPz/Hfbbef//9WPYfQvTjz7WqlCtXTualS5eWuTufuJa4tHCtUu7n6lpsnDVr\n1sjctbWFEMK5554baQzHNevMmDFD5onuLZ0SJUrIPEuWLDJ3P4+oXNOnu+8LIYRPPvkk0hjZs2eX\nuTv+XPtaotYqx7ULunNKnTp1Io+RXgoWLCjzZ555Rubu9xZno6e7FqxYsSLSGO4a++6778q8Xbt2\nMnffCULw95zJ9uGHH8r87rvvttuk5XuV4hp/69WrJ3N3fnC+/PJL+9o777wj8wULFsjcfb+I65of\nJ9da/Morr8jcNYO69kLXChlCCFOmTJF5pUqV7DbHi7/MAQAAAAAASCEs5gAAAAAAAKQQFnMAAAAA\nAABSCIs5AAAAAAAAKeSYizmDBg0KZcuWDRdddFG4+eabw8GDB8P27dtD3bp1Q6lSpUK9evXsQwEB\nAAAAAAAQrwxHjx496l5cu3ZtqF27dli2bFnInDlzaNmyZWjUqFH48ccfQ968eUPv3r3DkCFDwo4d\nO2QbVIYMGSJNxjUlPfvss3abRo0aRdqXe9J43759Ze4ajpyzzz5b5ome6r1jxw6ZHzx4UObuieKu\nFSuup6uH4Nu6nnrqKZm7J6///PPPMt+0aZPM77vvPpknegJ+XNzTyXv06CHzpk2bytw98T2ExMd4\nHLJmzSpz1/jk2lZCCCFz5swy37hxo8yHDx8uc3ccR+Vaifr162e3yZRJF/m5FgnXCLd9+/ZjzO6f\ne+2112TumipWr14tc/celi9fnraJxcA1mPTu3VvmrhVk69atMk/U9BLX8RdVqZh+cS8AABidSURB\nVFKlZN6xY0eZV6hQwe7rwIEDMn/iiSdk7trronKtNy+99JLdxrVy7dmzR+Z//fWXzFetWiVzd55x\n+0nkmmuukblrbnHXRfeec+TIIfPbb79d5lEbZhJx1zP388uXL5/M//77b5lPmzYt0v5DCGHp0qX2\ntSgaNmwo8+uvv17mF110USzjhuDP064lJaq0fOZcY5Hbl2swdG1g7jyTljarFi1ayNzdm7t7Qnf8\npSf3vcA1ELn2THcu2717t8zd9SGEEL799luZDxkyROb79u2z+1Ju+P/au/PoKur7jeOfC0FFKQfZ\nkkBCg0UORIHGplREqpZNXGhYTMGFiFBQapEKx8pSAbWCrQqUxYNoaSxHDa0KVEpEK6BQYgoNWkFB\nKCCEkLJaCLLn98/v0EP7eW6ceC/JyPv1V30uM/NNcu93Zr5N5snOdvPRo0e7eevWreW+rr/+ejcv\nKCgINKagGjRo4ObPPfec3Eb9rFWbnzqGascbNGiQmwe9nhkxYoR87fbbb3fzxMREN3/llVfcfNy4\ncW4etPU0llSb1fz58938H//4h5urNqto96SqJWzXrl1yG4+3bBP1N3Pq1q1rtWrVsiNHjtjJkyft\nyJEj1qRJE1u0aJHl5OSYmVlOTo6swgUAAAAAAEBsRV3MqV+/vo0cOdKaNWtmTZo0sXr16lnXrl2t\ntLT0zApdYmKilZaWnpPBAgAAAAAAnO+iLuZs2bLFpk6datu2bbNdu3bZ4cOHbd68eWf9m0gkEvjP\nqQAAAAAAAFA5URdz1qxZY9dcc401aNDAEhISrHfv3rZ69WpLSko682yTkpIS+TfVAAAAAAAAiK2o\nizmtWrWygoIC++KLL6y8vNzefvttS09Pt1tvvdVyc3PNzCw3N9eysrLOyWABAAAAAADOd36ly/9r\n166dDRgwwDIzM61GjRp21VVX2ZAhQ+zQoUOWnZ1tL7zwgqWlpcmnQAMAAAAAACC2olaTf+Wd8ywd\nAAAAAACASgtcTQ4AAAAAAIDqhcUcAAAAAACAEGExBwAAAAAAIERYzAEAAAAAAAgRFnMAAAAAAABC\nJGo1eZilpaW5eatWrdy8W7dubp6SkuLm2dnZlRpXEBdeeKGb//znP3fz9PR0N8/IyHDzlStXuvmg\nQYPkmNT39a677nLzQ4cOyX15vvGNb7h5ixYt3Hz//v1yXz/72c8CHTtWWrZs6eYTJkyQ2xw8eDBQ\nfuzYMTefOHGim//yl79085EjR7q5eu+Zme3ZsyfQNjfddJObr1q1Sh4jiNGjR7v5gAED5DaXXHJJ\noGN8/vnnbr5x40Y3f+qpp9y8oKAg0HFjqVatWm7esWNHN+/QoYObd+7cWR5jxowZbr5gwYIKRne2\niy66yM0HDhzo5tdff72bN2vWTB6juLjYzZcuXermzz33nNxXPGVmZsrXpk2b5uZ169Z181OnTrn5\n4MGD3XzNmjUVjC5+HnjgATfv3r27mx85csTN//Wvf7n5sGHDKjcwx29/+1s3b9KkiZvn5eW5+dy5\nc2M2plhJSkpy8wcffNDNN23a5OY/+tGP5DHUOUW9L9XP+tlnn3XzW265xc1Xr17t5sOHD3dzM7Pd\nu3fL18JOXQerOX/IkCFuvnbtWjfv0qVL5QYWgLoeyMnJcXN1Dblv3z43r127tjx2rL6+9u3bu/mo\nUaPc/LLLLnPzRo0ayWOoz/WSJUvcPCsrS+7L06NHDze/+OKL3Tw1NVXua+zYsYGO/dZbb7n5I488\n4uabN2928xo1/N93UPdgZvqzUrNmTTdX9047d+6Ux/i6uvfee9388OHDcpt58+a5+UMPPeTm6tq/\nadOmbq7el8ePH3fzhAS9rKLuw9S9eBD8Zg4AAAAAAECIsJgDAAAAAAAQIizmAAAAAAAAhAiLOQAA\nAAAAACHCYg4AAAAAAECIRMrLy8vjtvNIJF67PqNOnTpurp5Q/9FHH7m5evr+4sWL3TxoO0tlXH31\n1W6uGp8+/PBDN1dPZN+7d6+bnzhxQo5JNTKNHz9ebuNRbRTvvfeem6tWrJdeekke4/XXXw80JkW1\nFzz22GNurp5+rpofzHRbUlWJ1malmrRUg5P690EbChTVyLVixQq5TdCWnvr167u5ajgqKytzc9Wo\nEUvJyclurhob/vnPf7q5alNQbRRmek7p06ePm/fr18/Nn376aTdXTTlvvPGGm+/YscPNzXR7weOP\nPx4onzRpkjxGLKhmLzOzBg0auHnz5s3d/MUXX3Rz1f4xdOjQCkYXP3/84x/dXH3mVAtZVVLNW/n5\n+W6uGmZKS0tjNqagevbs6ebqOkG14dSrV08e48CBA26uGvVUK+CiRYvcXLXgqebOaGMdM2aMfC3s\npkyZ4uZq3lVtbapdsFOnTpUaVxDqlubOO+9084ULF7p5tAadeFOtsWo+Wb58uZur9kIz/XWrc6lq\ncFKNv+vXr3fzH/zgB26u7l/MzK655ho3v+CCC9xcnTNVA9Gbb77p5jNnznTzaE1T6npAtVyp6wo1\n94WJum5etmyZm6u26WeeeUYeQ7XXVZVo93njxo1z82itqx5vjuM3cwAAAAAAAEKExRwAAAAAAIAQ\nYTEHAAAAAAAgRFjMAQAAAAAACBEWcwAAAAAAAEIkoaoH8FXNmDHDzVXb1JNPPunmqoVl+/btlRtY\nAFlZWW6u2pKuvfZaNz8XjUgbN24M9O9vvPFGN1etVUePHnXz06dPBzpuLKmWFPXzqY6tKkGdPHlS\nvqZaq+655x43V80MsaKajypDNb/17dvXzdu2bevm3//+92M2pqBKSkrcPD093c3VHKfa9ObMmSOP\nrZoZFPW5Vg06qnUwllTzTXFxcVyPe8kll7h53bp15TYtW7Z080ceecTN1c9atRSeC6qR8NVXX3Xz\n6ji/qu+3aq9QjSeqqVK1OZrpZshYUQ1RtWrVcvNvf/vbbj5t2jR5jNzcXDdXrVXK8OHD3Xzfvn1u\n/re//c3NozWSfJ2NGjXKzWfPnu3mqp2qffv2MRtTUKtXr3bz6thapWzbts3NVRuhuhaN1kbYv39/\nN1dtVopqmlL3YJU5h0drJo0n9TWo9rBo1PvyyiuvdHPVBKWamquSGusnn3zi5uoaVTWlRmtlqyqq\nsW/ixIlyG9UIFwv8Zg4AAAAAAECIsJgDAAAAAAAQIizmAAAAAAAAhAiLOQAAAAAAACHCYg4AAAAA\nAECIRMrLy8vjtvNIJCb7Ue0IZmZFRUVurhoeVCtEvXr13Lxhw4ZuHvQJ+KoZxkw/5fy1115z8/ff\nf9/NVSNEaWlpBaP78tQTvJctW+bmqi1CtXao7/f69evdPD8/383NdMvRiRMn3PyCCy5w8y1btrj5\n3Xff7eZ/+ctf5JjCItq0sGfPnkD7ysvLc3PVAHPgwIFA+68M9fR91VLQpk0bN1fv7zfffNPNp0yZ\n4uaxbChQc80NN9wQKFeNXP369ZPHVu0P1dEVV1zh5mouS0pKcvNYNe2peUM1rJnp+VjNcc2bN3fz\neDd1mZnVqVPHzdU5+dlnn3XzWbNmuflHH31UuYHFgGp0UblqlrvsssvcPFojoGoDGzNmjNwmiMmT\nJ7u5mh9UE0+jRo3kMVSTW+/evd189+7dcl9BqEYu1aRmZvbBBx+4ubreOHjwYPCBxVnnzp3dXL2X\n1Hnx9ttvd/OqbJ9RTVpPPPGEm3fv3t3Nq/JrUNeix44dc/PKfB5UK2nQdlDVijVy5Eg3v+6669y8\nrKws0HGrK9X2OXjwYDdfuXKlm9euXdvNVSPSubjvUO9Lde7dunWrm6vrkM8++8zNu3Tp8iVG99Wo\nVjHVPHnLLbcEys30vX7Q63/v/ozfzAEAAAAAAAgRFnMAAAAAAABChMUcAAAAAACAEGExBwAAAAAA\nIERYzAEAAAAAAAiRULRZJScny9dUo8LixYvdXD0Ve/To0W7eoUMHNw/aYKKaPMzM2rVrF2hfGRkZ\nbn7bbbe5uXp6fGVkZma6uWr1eeedd9xcNW8p6mseMmSI3Gbq1KluPnbs2EDHbty4sZtPmDDBzbt1\n6+bmhYWF8hjqyf8lJSXRBxcnF198sXxNNcVdeOGFbq4aL1RbjWroKSgokGMKSjUENWvWzM1V249q\nQ7nvvvvcvEePHm4ey6f1T58+3c0vuugiN3/33XfdvGXLlm6u3t9mZt/73vcqGF31oRr1VHvdgAED\n3Dxao168NWjQwM1Vs4VqQhw6dGjMxhRUzZo13Vy1QowYMcLNVWtQ37593bwq22oUNS9NmzZNbnPV\nVVe5eceOHWMypnNh4cKFbq5+Rqrl6ly499573Xz8+PFu/pvf/MbNVZuQaqKrjB07dri5aqVZu3at\nm6vzVkJCgptXx8+WOier9secnJx4Duec6NWrl3xNXR9/85vfDHQM1USnWp1U+9XcuXPlMR566CE3\n37t3bwWjO/datGjh5qpBVd0XqH+v7m1VO+xbb73l5pWh2hn79Onj5uq+Rp3zT5486eaVWUtQ13d/\n/vOf3Vzd16gmv0WLFrn5d77zHTmmgQMHuvnHH3/s5sOGDXNz2qwAAAAAAABCjsUcAAAAAACAEGEx\nBwAAAAAAIERYzAEAAAAAAAgRFnMAAAAAAABCJBRtVtGoJ35PnDjRzW+++WY3v+GGG9x8+/btgcaj\nnrr+q1/9KtB+olFNQ2vWrHHz9PT0mB1bSUlJcfOdO3fGZP8NGzZ089dee01us3//fjfPysqKyZiC\nuvPOO+VrqtlJPfn/jTfeCHRs9TUvWLAg0H5iqW7dum6uGrxUa0JlpKWlufm2bdtisn/VLKCedP/D\nH/4wJseNJdXAp568b6ab+dRnsTqaPXu2m6t2L3XuqEqq5U+1m6i2mjDJy8tz81dffdXN58+fH8/h\nRJWYmOjmzz//vJsnJSXJfam5Y9euXW6uGtDUnPjhhx+6edDWJdX8Z6bnFHV+V9dZivqay8rK3Pzo\n0aOB9m+m21OmTJni5qoRVX12VQOVyqM5fPhwoH0dO3bMzevVq+fmn3/+eeAxqdbVFStWBN5XEKpt\nbN++fW6u7i2iiff1hvq5rV+/Xm4zb948N6/M1xeEuu6744475DbDhw9389atW8dkTLGkWl1TU1Pd\nfPPmzYH237ZtWzefOXOmm3fq1CnQ/s+F2rVru7m6X1Tzd2WoOUu1YQYV7TynrjnUPQ9tVgAAAAAA\nAF9TLOYAAAAAAACECIs5AAAAAAAAIcJiDgAAAAAAQIiwmAMAAAAAABAiLOYAAAAAAACESEJVHHTU\nqFFuXlRU5OZ33XWX3Ndtt93m5rNmzXLzjIwMN69MlaLn2muvdXNVh2amq1NVNaKqUvz9739fwei+\nuq5duwbKDx065OaqXk/V/6oq13feecfNzcyeeeYZ+Zrn6quvdnNV46nqVE+dOuXmqgrSzGzJkiVu\nvnfvXjdv0qSJm6uKO1WBuXHjRjdXn0Uzs9zcXDdXP4uTJ0+6+a233urmGzZskMeOFVURqN4DjRs3\ndvNf//rXbt6iRQs3z87O/hKjO9ucOXPcXNWcr1q1ys1VPbh6b/z0pz9185SUFDc3M7v//vvd/NFH\nH5XbxMK3vvUtN7/77rvlNu3bt3fzbt26ufngwYMDjWnRokVufuONN7p58+bN3by4uFgeQ72fnnji\nCTdX58ug1Hlu5cqVMdm/mX5/jxgxws3VnLhgwYLAx65Rw///uRo1auTm3/3ud908JyfHzdPT0918\n7ty5bj59+nQ3N9O10cqECRPcXH2/VfX5nj173LxVq1Zurq7JzHQtrBprUKrKXOV/+tOf5L7ef/99\nN//rX//q5r/73e/c/A9/+IM8RpDjDh06VG6zbt06N1ef35dfftnNs7Ky3DxW181mZj/5yU/cXF1/\nrV27NibHVZXOb7/9tptXprr76aefdvM+ffq4uaoav/nmm938xRdfdPPnn39ejilWFeT9+/d38y1b\ntrh5WVmZm6u51cysTp06wQdWRZ566ik3//vf/+7m27dvD7T/zp07u/m///3vQPt56aWX5GvqPkJd\nxwWdBx5++GE3Pxf3sOpcoyrF1bVlx44d3TzaNae6txk3bpzc5sviN3MAAAAAAABChMUcAAAAAACA\nEGExBwAAAAAAIERYzAEAAAAAAAgRFnMAAAAAAABCJFJeXl4et51HIm4+depUN1dNUwUFBfIYqull\n8+bNFYwuPtQTsXv37i236dKli5s3bNjQzRcvXuzm0Z5cHytTpkxx89atW7u5atZRLVeqfaGwsNDN\njxw54uaVUbt2bTdXrUsdOnQItH/VYGKm287U9+mmm25y8+PHjwcak6LaK8x0Q0tmZqabJycnu7lq\nvunVq5ebHzhwQI4pKNWc8cADD7j5Z5995uaqRULllXHFFVe4+Y9//GM3V/NoaWmpm6tGs02bNrm5\namUwM1u/fr18LZ7Gjh3r5up7YabnJtUs98knnwQak/o8zJ49281Pnz4daP9mZu+++66bq/Yj1VwW\nlPqc9O3bV26jGuEUNeer89yKFSsC7T+axx57zM07derk5h988IGbq8Yi1XxUmfdArAwcONDNVQNa\nzZo13fzo0aNurq77zMyWLVtWwejiQzWVqHO+mVm7du3c/Morr3TzSy+91M3VtZRqV1JNgZMmTXJz\nM319pNq6fvGLX7i5areJpaSkJDdX3yd1vaYaY1RboGpKUnPo0qVL3Twa9dkaM2aMm9evX9/NVcuo\n2o+aQ2NJ3ac8/vjjbq6ab6M1Iarz+86dOysY3bmnGnnVOVM1+37xxRdurlrtZsyY4ebquk81rpqZ\njR8/3s3Vtahqs1LvDfWzVnOcagiujNTUVDdX5yDVWqzOZ+qazMzsxIkTFYzuy/GWbfjNHAAAAAAA\ngBBhMQcAAAAAACBEWMwBAAAAAAAIERZzAAAAAAAAQoTFHAAAAAAAgBCpkjYrAAAAAAAAVIw2KwAA\nAAAAgJBjMQcAAAAAACBEWMwBAAAAAAAIERZzAAAAAAAAQoTFHAAAAAAAgBCJ62LOddddF8/dAwAA\nAAAAfG2pdZW4VpMDAAAAAAAgtvgzKwAAAAAAgBBhMQcAAAAAACBE4r6Yk5+fb61atbLLL7/cnnzy\nyXgfDgBiJi0tzdq2bWsZGRnWvn17MzPbv3+/de3a1Vq2bGndunWzgwcPVvEoAeA/7rnnHktMTLQ2\nbdqcyaLNW5MmTbLLL7/cWrVqZUuXLq2KIQPA//DmsgkTJlhKSoplZGRYRkaGLVmy5MxrzGU4H8V1\nMefUqVN2//33W35+vm3YsMFefvll+/jjj+N5SACImUgkYsuXL7eioiIrLCw0M7PJkydb165dbdOm\nTda5c2ebPHlyFY8SAP5j4MCBlp+ff1am5q0NGzZYXl6ebdiwwfLz823YsGF2+vTpqhg2AJzFm8si\nkYg9+OCDVlRUZEVFRdajRw8zYy7D+SuuizmFhYXWokULS0tLs1q1alm/fv1s4cKF8TwkAMTUfz8j\nftGiRZaTk2NmZjk5ObZgwYKqGBYAuDp16mSXXnrpWZmatxYuXGj9+/e3WrVqWVpamrVo0eLMwjUA\nVCVvLjP73+syM+YynL/iuphTXFxsqampZ/47JSXFiouL43lIAIiZSCRiXbp0sczMTJszZ46ZmZWW\nllpiYqKZmSUmJlppaWlVDhEAKqTmrV27dllKSsqZf8d1GoDqbvr06dauXTsbNGjQmT8ZZS7D+Squ\nizmRSCSeuweAuFq1apUVFRXZkiVLbObMmfbee++d9XokEmGeAxAqFc1bzGkAqqv77rvPtm7dauvW\nrbPk5GQbOXKk/LfMZTgfxHUxp2nTprZjx44z/71jx46zVk0BoDpLTk42M7NGjRpZr169rLCw0BIT\nE2337t1mZlZSUmKNGzeuyiECQIXUvPXf12k7d+60pk2bVskYAaAijRs3PrMgPXjw4DN/SsVchvNV\nXBdzMjMz7dNPP7Vt27bZ8ePHLS8vz3r27BnPQwJATBw5csQOHTpkZmZlZWW2dOlSa9OmjfXs2dNy\nc3PNzCw3N9eysrKqcpgAUCE1b/Xs2dNeeeUVO378uG3dutU+/fTTM819AFDdlJSUnPnfr7/++pmm\nK+YynK8S4rrzhASbMWOGde/e3U6dOmWDBg2y1q1bx/OQABATpaWl1qtXLzMzO3nypN1xxx3WrVs3\ny8zMtOzsbHvhhRcsLS3N5s+fX8UjBYD/6N+/v61YscL27t1rqamp9uijj9rDDz/szlvp6emWnZ1t\n6enplpCQYLNmzeJPEwBUC/89l02cONGWL19u69ats0gkYs2bN7fZs2ebGXMZzl+Rcu+R4AAAAAAA\nAKiW4vpnVgAAAAAAAIgtFnMAAAAAAABChMUcAAAAAACAEGExBwAAAAAAIERYzAEAAAAAAAgRFnMA\nAAAAAABChMUcAAAAAACAEGExBwAAAAAAIET+D6x73gK90iRFAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import nn_utils as nn\n",
"nn.show_images(x_train[:200], (10, 20), scale=1)\n",
"nn.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Placeholders and Variables"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#Placeholders\n",
"X = tf.placeholder(\"float\", shape=(None, 64))\n",
"Y = tf.placeholder(\"float\", shape=(None, 1))\n",
"\n",
"#Varialbels\n",
"W = tf.Variable(np.random.randn(64, 1).astype(\"float32\"), name=\"weight\")\n",
"b = tf.Variable(np.random.randn(1).astype(\"float32\"), name=\"bias\")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"TensorShape([Dimension(None), Dimension(64)])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X.get_shape()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Logistic Regression Model \n",
"\n",
"We will consider linear model for classification. Note that the model is linear in parameters.\n",
"\n",
"$$y(\\mathbf{x}, \\mathbf{w}) = \\sigma (\\mathbf{x}^T \\mathbf{w} + b)$$"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TensorShape([Dimension(None), Dimension(1)])\n"
]
}
],
"source": [
"logits = tf.add(tf.matmul(X, W), b)\n",
"output = tf.nn.sigmoid(logits)\n",
"\n",
"print output.get_shape()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cross-Entropy Cost"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cross-Entropy cost = $t * -\\text{log}(y) + (1 - t) * -\\text{log}(1 - y)$ \n",
"\n",
"Cross-Entropy cost = $t * -\\text{log}(\\sigma(x)) + (1 - t) * -\\text{log}(1 - \\sigma(x))$ \n",
"\n",
"where $ \\sigma(x) = {1 \\over {1 + e^{-x}}}$\n",
"\n",
"### Problem: This cost will give rise to NaN when $x \\rightarrow -\\infty, \\infty$"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"1.0"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def sigmoid(x):\n",
" return 1.0 / (1.0 + np.exp(-x))\n",
"sigmoid(100)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How not to do Cross Entropy"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"4.53988992168e-05\n",
"nan\n",
"inf\n",
"inf\n"
]
}
],
"source": [
"#This doesn't work!\n",
"def xentropy(x, t):\n",
" return t*-np.log(sigmoid(x)) + (1-t)*-np.log(1.0 - sigmoid(x))\n",
"print xentropy(10, 1)\n",
"print xentropy(-1000, 0)\n",
"print xentropy(1000, 0)\n",
"print xentropy(-1000, 1)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-1.11022302463e-15\n",
"-1.11022302463e-15\n",
"34.4342154767\n",
"34.5387763949\n"
]
}
],
"source": [
"#This kind of works!\n",
"def hacky_xentropy(x, t):\n",
" return t*-np.log(1e-15 + sigmoid(x)) + (1-t)*-np.log(1e-15 + 1.0 - sigmoid(x))\n",
"print hacky_xentropy(1000, 1)\n",
"print hacky_xentropy(-1000, 0)\n",
"print hacky_xentropy(1000, 0)\n",
"print hacky_xentropy(-1000, 1)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-0.0\n",
"-0.0\n",
"inf\n",
"inf\n"
]
}
],
"source": [
"#This kind of works!\n",
"def another_hacky_xentropy(x, t):\n",
" return -np.log(t*sigmoid(x) + (1-t)*(1-sigmoid(x)))\n",
"print another_hacky_xentropy(1000, 1)\n",
"print another_hacky_xentropy(-1000, 0)\n",
"print another_hacky_xentropy(1000, 0)\n",
"print another_hacky_xentropy(-1000, 1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How to do Cross Entropy\n",
"Cross-Entropy = $x - x * t + log(1 + e^{-x}) = max(x, 0) - x * t + log(1 + e^{-|x|}))$"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.0\n",
"0.0\n",
"1000.0\n",
"1000.0\n"
]
}
],
"source": [
"def good_xentropy(x, t):\n",
" return np.maximum(x, 0) - x * t + np.log(1 + np.exp(-np.abs(x)))\n",
"print good_xentropy(1000, 1)\n",
"print good_xentropy(-1000, 0)\n",
"print good_xentropy(1000, 0)\n",
"print good_xentropy(-1000, 1)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEPCAYAAABLIROyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHkBJREFUeJzt3XmYVOWVx/Fv02yyaIuAgBIbNSqJmlYnSFikEVAEBGQT\nF7R1QjSiMxqDBp0JMS4D4iBqJBEQGyVo2GURFJcGFCMugJqAwJhGkCWooGBAQXr+ONV29V7VdW+9\nt+r+Ps9TT9ctqquO5ynr7XvOfd8XRERERERERERERERERERERERERERESpkK7AI+iHpsHLAeWAfM\nBY5xEJeIiPisM3AOpQeAHkCtyP0xkZuIiDhQq/qn1NhKYE+Zx5YBRyL33wJO9PH9RUSkCn4OANW5\nHnjB4fuLiISaqwHgbuBbYIaj9xcRCb3aDt4zD+gFdKvsCa1atSravn170gISEUkT/wecGuuTk30G\n0BMYCfQDDlb2pO3bt1NUVKSbR7fRo0c7jyFdbsql8hnkG3BKPF/Ifg4AzwKrgNOBrVjN/zGgEdYM\nXgNM9PH9JaKwsNB1CGlDufSW8umWnyWgKyp4bKqP7yciInFweRWQJEleXp7rENKGcukt5dOtDNcB\nVKIoUs8SEZEYZWRkQBzf6zoDCIGCggLXIaQN5dJbyqdbGgBEREJKJSARkTShEpCIiMREA0AIqM7q\nHeXSW8qnWxoARERCSj0AEZE0oR6AiIjERANACKjO6h3l0lvKp1saAEREQko9ABGRNKEegIiIxEQD\nQAiozuod5dJbyqdbGgBEREJKPQARkTShHoCIiMREA0AIqM7qHeXSW8qnWxoARERCSj0AEZE0oR6A\niIjERANACKjO6h3l0lvKp1saAEREQko9ABGRNKEegIiIxEQDQAiozuod5dJbyqdbGgBERELKzx7A\nVKA38E/grMhjTYC/ACcBhcAQYG8Fv6segIhInILUA3gK6Fnmsd8Ay4DTgFcixyIi4oCfA8BKYE+Z\nx/oC0yL3pwH9fXx/iVCd1TvKpbeUT7eS3QM4HtgVub8rciwiIg74PQ8gG1hISQ9gD3Bs1L9/gfUF\nylIPQEQkTvH2AGr7F0qFdgEtgJ1AS6xBXKGOHfPo0SMbgKysLHJycsjNzQVKTht1rGMd6zjMxwUF\nBeTn5wOQnZ1NvJJ9BvAg8DkwFmsAZ1FxI7ioRYsiJk+GPn18jjAECgoKvv/wSGKUS28pn94K0lVA\nzwKrgNOBrcB1wBigB7ARuDByXKHnn4frroM33/QxQhGREAv0WkBLltgg8Npr0Lat65BERIItSGcA\nCbvkEnjwQejZE7Ztcx2NiEh6CfQAAHDNNTBihA0Ge8rOKpCYFDeNJHHKpbeUT7cCPwAAjBwJ3btD\nv35w4IDraERE0kOgewDRjhyBq66Cb76BWbMgM9NRZCIiAZVWPYBotWpBfj589ZWVhDRPTEQkMSkz\nAADUqwdz58Lq1fD737uOJnWozuod5dJbyqdbyZ4JnLCjj4YlS6BDB2jZEn7xC9cRiYikppTpAZS1\neTNccAFMnAj9taaoiEjg1wLyzKmnwsKFdnlo06bQqZPriEREUktK9QDKOu88mD4dBg6EDz90HU1w\nqc7qHeXSW8qnWyk9AABcdBGMHw+9esEnn7iORkQkdaRsD6Cs8eNhyhR4/XVoUtEOAyIiaS7eHkDa\nDABgM4ZXrYJly6BBAx+iEhEJsLSdCBaLsWPh5JNh6FA4fNh1NMGhOqt3lEtvKZ9updUAUKsWPPmk\nLRdx442aLSwiUpW0KgEV278funa1ZaTvvdfDqEREAiw08wCq0qgRLF5scwNatoSbbnIdkYhI8KRV\nCSha8+awdCncfz/MmeM6GrdUZ/WOcukt5dOttDwDKHbyybBoEVx8sc0W7tLFdUQiIsGRlj2Asl59\n1a4MevllOPtsz15WRCRQQn0ZaGUuvBAee8xmC2/Z4joaEZFgCMUAAHD55XDHHVYO+uwz19Ekl+qs\n3lEuvaV8uhWaAQDgP/4DLrsM+vSBr792HY2IiFuh6AGUfmG47jrYvRvmz4c6dXx5GxGRpFMPoBoZ\nGTB5st0fPlyzhUUkvEI3AID91T9zJmzYAHff7Toa/6nO6h3l0lvKp1uhHAAAGja0OQJz59oVQiIi\nYRO6HkBZhYW2ZMT48TBkSFLeUkTEF1oLKE7Z2fDCC9C9u80WvvBC1xGJiCSHqxLQKOBvwAfADKCe\nozgAmx08c6bNFl6zxmUk/lCd1TvKpbeUT7dcDADZwHDgXOAsIBMY6iCOUnJzYeJEmyPwj3+4jkZE\nxH8uegBNgDeB9sA+YB7wCPBy1HOS1gMoa+JEmDAB3ngDmjVzEoKISI2kwjyAL4D/BT4BtgN7Kf3l\n79RNN1kzuHdv21hGRCRduWgCnwLcipWCvgRmAVcBf45+Ul5eHtnZ2QBkZWWRk5NDbm4uUFI39Ou4\nW7cC1qyBQYNyWbAAVq3y9/38Pp4wYUJS85fOx9E16yDEk+rHymfi+cvPzwf4/vsyHi5KQJcDPYCf\nR46HYeWgEVHPcVYCKnb4MAwYAMccA9Om2X7DqaqgoOD7D48kRrn0lvLprXhLQC4GgJ9gf+3/FDgI\n5AOrgcejnuN8AAD417/s8tBOneDBB11HIyJStVToAawDngbeAd6PPDbJQRzVatDAZgsvWgQPP+w6\nGhERb8UyAGT68L4PAj/GLgO9Fjjkw3t4okkT21t4/HiYMcN1NDUTXWeVxCiX3lI+3YplANgEjAN+\n5HMsgfWDH8CSJXDbbbBsmetoRES8EUut6GhsolYedjYwFXgW+Mq/sILRAyhr5UprDC9dCued5zoa\nEZHS/G4C52IN3GOxyzfvBTbH+RqxCOQAALaJzE03wYoVcOqprqMRESnhRxO4NtAPmA9MwCZxnQws\nBF6IP8TU1r8/jB4NPXvCrl2uo4mN6qzeUS69pXy6FctEsI1AAda4XRX1+Gygiw8xBd4NN8COHdCr\nFxQUQOPGriMSEYlfLKcKjbE1e5IpsCWgYkVFcOON8PHHsHgx1K3rOiIRCTs/SkDNsXLPZ8Bu4Hms\nBBRqGRm2cFyjRpCXB0eOuI5IRCQ+sQwAM4CZQEugFdb8fdbPoFJFZqbNDdi6FW6/PbgbzKvO6h3l\n0lvKp1uxDABHAc9gk7UOAdOB+n4GlUqOOgoWLLD5AQ895DoaEZHYxVIrGost2Vz8V//l2GWgxavj\nfOFDXIHvAZS1bRt07Aj33QfDhrmORkTCyI95AIVAZd/GRfjTD0i5AQBg/Xro2hXy8+0yURGRZPKj\nCZwNtKnkFvpmcLS2bWHuXDsDWL3adTQlVGf1jnLpLeXTrVgGgLrAfwJzsGv/bwHq+BlUKuvQAaZO\nhX79YONG19GIiFQullOFJ7EJY9Mizx8GHKZkQxc/pGQJKNqUKfDAA7a3cMuWrqMRkTCItwQUy0zg\nnwJnRx2/Qsk6/lKJn/8cdu6ESy6B5cttZzERkSCJpQR0GIhe9uyUyGNSjbvvtt3E+veHb75xF4fq\nrN5RLr2lfLoVywAwEngVWB65vQr82s+g0kVGBjzyCDRtao1hzRYWkSCprlaUiTWAJwKnRx77CNvL\n108p3wOIdvCgXRZ69tk2IGS42IlZRNKeH/MA3sb6AMmUVgMAwN69cMEFcMUVMGqU62hEJB35MQ/g\ndeAPQGfgXOC8yE+JQ1aW7SQ2aRI89VRy31t1Vu8ol95SPt2K5Sqgc7AZv78v83hX78NJb61a2SDQ\npQs0awZ9+riOSETCLJZThZOBj2N4zEtpVwKK9tZb9uW/cCG0b+86GhFJF36UgGZX8NisWN9Ayjv/\nfJg2zS4P3bDBdTQiElZVDQBtgYFAFjAgcn8AkIeWg05Yr14wZoxdHfTpp/6+l+qs3lEuvaV8ulVV\nD+A04FLgmMjPYvuA4X4GFRZ5eTZbuGdPWLnSGsUiIskSS62oA6U3g0+GtO4BRCsqgltvhbVr4cUX\nob7OrUSkhvyYB9Ac+4s/m5IzhiLg+jhji0doBgCwGcJXXAGHD8PMmbbVpIhIvPxoAj8PHA0sAxZH\n3cQjtWrB00/bZLFbbvF+b2HVWb2jXHpL+XQr1j2B78Q2hp8duc1J8H2zIq+zHvg7EPqLIevVg3nz\n4M03bVtJERG/xXKqcB/wJt7+1T8NW1huKlZWagh8GfXvoSoBRdu50zaVGTUKhqvVLiJx8KMHsB9o\nAHwLHIo8VoSVhWriGGANVW8nGdoBAGDTJls36E9/sp3FRERi4UcPoFHkefWBxpFbTb/8wfYS3g08\nBbwHTMYGGIn44Q9hwQLbVOaNNxJ/PdVZvaNcekv5dKuqeQBXA9Mj9zsC0V9FN2MLxNX0Pc+NvMbb\nwATgN8Bvo5+Ul5dHdnY2AFlZWeTk5JCbmwuUfGjS/Xj69FwGDIAxYwpo06bmr7d27dpA/PfoWMc6\n9va4oKCA/Px8gO+/L+NR1anCGmwhuLL3KzqORwusp9AmctwJGwCil0YLdQko2vTpcNdddibQurXr\naEQkyPzYE9hrO4Gt2EzjjUB34G8O4kgJV19derZwkyauIxKRdBFLD8APtwB/BtZhG84/4CiOlPDr\nX9sA0LcvHDgQ/+8XnzJK4pRLbymfblU1AJwBfBC5nR51v/g4EeuwXcZ+gi0w92XVT5dx4+Ckk0pm\nDIuIJKqqWlF2Nb9b6F0Y5agHUIFvv7V9BLKz4YkntLewiJTmxzyAaH2ARXH+Tk1oAKjEvn3QtSv0\n7g333OM6GhEJEj/mAUS7N87ni8caN4YXXoAZM2yiWCxUZ/WOcukt5dMtF1cBSYKaN7e9hTt3tvsD\nBriOSERSUbwloHbAaj8CKUMloBi8955dHTR7ti0dISLh5kcJaAglSz9cDMzDZvKKY+eea6WgwYPh\ngw9cRyMiqSaWAeC/ga+wGbvdgCeBP/oZlMSue3d45BHbY3jLloqfozqrd5RLbymfbsUyAHwX+dkH\nW7htEVDXt4gkbkOHwu23Wzno889dRyMiqSKWWtFi4FOgB7b+z0HgLWwSl1/UA6iBO++EFSvglVeg\ngdZXFQkdP+YBNAR6Au8Dm4CWwFnASzWIL1YaAGrgyBHIy4M9e2x3sdq6xkskVPxoArfAzgI2AV2x\npnAyrgSSONWqBU8+aUtF3HBDyd7CqrN6R7n0lvLpViwDwFzgMHAq8ARwIjDDz6Ck5urUgVmz7Kqg\n//ov19GISJDFcqpQvPb/HcAB4DES2w8gFioBJWj3bujUCW6+GW65xXU0IpIMfuwH8C1wJXANcGnk\nsTpxRyZJ1ayZzRbu1AlatLC5AiIi0WIpAV0P/Ay4H/gHtpn7M34GJd5o0wYWL4bhwwt47TXX0aQH\n1ay9pXy6FcsA8Dfg18CHwJnYbl5j/QxKvJOTA6NHw+WXw7p1rqMRkSCJpVaUC0wDiueZ/gC4Flju\nU0ygHoDnZs2CW2+F11+3MwMRST9+9ADGAxcBH0WOTwOeQ+sBpZTBg2HXLrj4Yttgvlkz1xGJiGux\nlIBqU/LlD7aRu6YYpZDiOuvNN9tA0Ls37N/vNqZUpZq1t5RPt2IZAN4FpmCloK6R++/4GJP46L77\n4KyzYNAg22JSRMIrllpRPeBmoGPkeCUwEfjGr6BQD8BXhw/DwIHQsCFMn24ziEUk9Xm9FlBt7Oqf\nMxKIqSY0APjswAHrB+Tk2HLS2mBeJPV5vRbQYaz+f1ICMYljFdVZjzoKFiyA5cvh/vuTH1OqUs3a\nW8qnW7E0c5tgcwFWA19HHisC+voVlCRHVpbNFu7Y0a4KuuEG1xGJSDJVdarwQ+B4ILPM8zoDO7Bm\nsF9UAkqizZttT+FHH7XmsIikJi97AIuBUdg+ANHOxpaFuLTcb3hHA0CSrVljPYHnnoMLL3QdjYjU\nhJc9gOMp/+VP5DHNJU0hsdRZzzkHZs607SXffdf/mFKVatbeUj7dqmoAyKri3+p7HYi4l5sLTzwB\nffrApk2uoxERv1V1qvAc8Cowqczjw4HuwOUJvncmNqFsG+XLSSoBOTRlil0Z9MYb0KqV62hEJFZe\n9gBaAPOw/QCKiwLnYRPDLsMawYn4VeT1GlP+iiINAI79z//AjBm2yfyxx7qORkRi4WUPYCfQAbgH\nKMT2ArgHaE/iX/4nAr2wK4k0BclnNamz/uY30KMHXHop/Otf3seUqlSz9pby6VZ1E8GKsDLQo9hW\nkK969L4PAyOBIx69nngsIwMeesiWjh4yBA4dch2RiHjNxV/ffYBLgBHYAnO3U0EP4NprryU7OxuA\nrKwscnJyyM3NBUr+atCx/8eHDkGnTgVkZcGSJbnUqhWs+HSs4zAfFxQUkJ+fD0B2djb33HMPeLgW\nkB8eAIZhy0zUB44G5mB7DhdTDyBAvv7aykEdOsC4cVo3SCSovF4LyA93Aa2xuQRDsbLSNVX+hiSk\n+C+GmmrYEBYtsmUjxo3zJqZUlWgupTTl060gbOyiP/VTQJMm8OKLtm5Q06Zw/fWuIxKRRAX1ZF4l\noID66KOSCWN9tRygSKB4vR+AKxoAAuydd6BXL5g92xaRE5FgSIUegCSZ13XWf/s3myQ2aBCsW+fp\nSweeatbeUj7d0gAgNdK9Ozz+uJ0JaN0gkdSkEpAkZPJkeOABWLkSTjzRdTQi4RZvCSgIVwFJChs+\nHPbutXkCK1bYzmIikhpUAgoBv+usI0fCgAHQsyd8+aWvb+WcatbeUj7d0gAgnrjvPmjf3i4NPXDA\ndTQiEgv1AMQzR47AsGFWEpo3D+rWdR2RSLhoHoA4degQDBxoy0dMnw6Zma4jEgkPzQOQcpJZZ61T\nx/YW3rkTRoyAdBvHVbP2lvLplgYA8Vz9+rBgAbz3Howa5ToaEamMSkDim88/t6Uihg2zHcZExF+a\nByCBcdxx8NJL0LkzZGXBjTe6jkhEoqkEFAIu66wnnADLltllos8+6ywMz6hm7S3l0y2dAYjvTjnF\nNpPp3h0aNbKN5kXEPfUAJGlWr4Y+feDPf7alI0TEW7oMVAKrXTuYOxeuvBKWL3cdjYhoAAiBINVZ\nO3WC556DwYPhr391HU38gpTLdKB8uqUBQJKuWzeYNg369bO5AiLihnoA4sy8efDLX8LLL8OZZ7qO\nRiT1aR6ApIzLLoODB+Gii+C11+D0011HJBIuKgGFQJDrrFdcYTuKde8OH3/sOprqBTmXqUj5dEtn\nAOJcXp7tIdCtm+0q1rq164hEwkE9AAmMhx+GiROtHKT9hUXipx6ApKzbboPDh6FrVw0CIsmgHkAI\npFKddeRI+MUvbBDYts11NOWlUi5TgfLpls4AJHBGjrSfOhMQ8Zd6ABJY48bBpEkaBERilQo9gNbA\n00BzoAiYBDzqIA4JOJ0JiPjLRQ/gEHAb8GOgPTACaOsgjtBI5Tpr0HoCqZzLIFI+3XJxBrAzcgPY\nD6wHWkV+ipQzciRkZECXLrZsRJs2riMSSQ+uewDZwHLsbGB/1OPqAUg5jz8OY8faDmNaNkKkvFTo\nARRrBMwG/pPSX/4iFRoxAho2tHLQ0qVw9tmuIxJJba4GgDrAHGA6ML+iJ+Tl5ZGdnQ1AVlYWOTk5\n5ObmAiV1Qx3HdjxhwoS0yV9eHhQWFtClC7z4Yi7t2iX3/aNr1kHIR6ofK5+J5y8/Px/g++/LeLgo\nAWUA04DPsWZwRVQC8lBBQcH3H550sWgRXHcdzJ5tvYFkScdcuqR8eiveEpCLAaATsAJ4H7sMFGAU\nsDTqORoApFqvvAJDh8Izz0DPnq6jEXEvFQaAWGgAkJisWgX9+8Mf/gBDhriORsQtbQov5UTXWdNN\nhw52VdCvfgWPPeb/+6VzLl1QPt3SWkCS8n7yE3j9dbj4YtixA+6/3+YNiEjVgvq/iUpAErfPPoPe\nvaFtW5g8GerUcR2RSHKpBCSh1bQpvPoq7N4N/frBfs0uEamSBoAQCFOdtWFDmD8fWraEzp29Xz8o\nTLlMBuXTLQ0Aknbq1IEpU+DKK6F9e3j7bdcRiQSTegCS1ubPh+HDba/hwYNdRyPir1RaC0jEd/37\nw0knWU9g40a46y5dISRSTCWgEAh7nfWcc+Cvf4Xnn7fJYvv21fy1wp5LrymfbmkAkFBo1QpWrIBj\nj4V27WC9dp8QUQ9Awuepp+COO9QXkPSjtYBEYvDeezBokPUIxo7VpDFJD5oIJuWozlreuefCO+9Y\nY7hDB/sZC+XSW8qnWxoAJLSaNIGFC21fgY4dYdIk0ImnhIlKQCJYU/iqq+DEE20SWfPmriMSiZ9K\nQCI10LatXSr6ox9BTg7Mm+c6IhH/aQAIAdVZY1O3LowZA3/5C4waBQMGwPbtpZ+jXHpL+XRLA4BI\nGZ07w9q18OMf214DEyfCd9+5jkrEe+oBiFThww/h5pthzx549NHkbkAvEi/NAxDxWFERzJoFI0fC\n+efDAw/Aqae6jkqkPDWBpRzVWROTkWFrCK1fD40bF9C+Pdx4Y/n+gMRPn023NACIxKhBAxg2DD76\nCBo3hjPPhFtvha1bXUcmUjMqAYnU0PbtMH48TJ1qS0rccQeccYbrqCTMVAISSZJWreChh2DTJttz\noEsXuOgiW3ZaVw1JKtAAEAKqs3qnolwedxyMHg2ffALXXGNzCU4+GX77WxscpHL6bLqlAUDEI/Xq\nwdVXw5tv2lnAvn3QqZOtM/Too7Bli+sIRUpTD0DER4cOwUsvwZw5sGiRlY369bOeQU6OtqcUb2ke\ngEhAffcdrFplZwfz59sZQpcudsvNtXWINCBIIlKlCdwT2ABsAu50FENoqM7qnURymZlpy0w89BBs\n3gxvvQW9e8O770LfvrYCad++8LvfwYIFsG1b+i9Prc+mW7UdvGcm8AegO/Ap8DawANAurT5Zu3Yt\nubm5rsNIC17mMjvbbtdea8dbt8Lbb9tuZX/6kw0MRUU23+C00+D00+122ml21VE67GKmz6ZbLgaA\ndsBmoDBy/BzQDw0Avtm7d6/rENKGn7ls3dpuAwbYcVGRzTX4+99t8tnGjbB0qd3fvh2aNbP9C1q3\nLv2zeXNo2tRuxx1nq5wGlT6bbrkYAE4AoudObgPOdxCHSKBlZMAJJ9itR4/S/3b4MOzYYWWirVvt\ntm2b7Wmwezd89pndvvgC6te3waBJEzj6aGjUyGYyV3Q76ii7mqlePfu9svejH6tXD2rXttJWZqb6\nF6nIxQCQ5lXN4CksLHQdQtoISi5r1y45Y/jZzyp/XlERfPVVyWCwb1/Fty1b7OeBA/DNN3Y7eLD6\n+999Z4PRkSM2AEQPCNH3yx4X39++vZA5c+x3y96g4scT+bey/16d6p7jxWt48T7t2tm8k3i5GLPb\nA7/DGsEAo4AjwNio52wGTkluWCIiKe//gECvVVsbCzIbqAusBdq6DEhERJLnEuAj7C/9UY5jERER\nERERFwYDfwO+A84t82+jsEljG4CLkhxXOvgddrXVmsitZ5XPlspoAqO3CoH3sc/karehpJypwC7g\ng6jHmgDLgI3AS0CWg7hq7AzgNOA1Sg8AP8L6BHWwvsFmtIhdvEYDv3IdRIrLxD572dhnUb2rxP0D\n+9KS+HUGzqH0APAgcEfk/p3AmOpeJEhfpBuwkausfsCzwCHsL4bN2GQyiY+u0k5M9ATGQ5RMYJTE\n6HNZMyuBPWUe6wtMi9yfBvSv7kWCNABUphVWvii2DZtMJvG5BVgHPEmKnRoGREUTGPU5TEwR8DLw\nDjDccSzp4HisLETk5/HV/UKyJ4ItA1pU8PhdwMI4XkeTycqrLLd3A38Efh85vhf4X+DfkxRXutBn\nznsdgR1AM+zzuwH7y1YSV0QMn9lkDwA9qn9KOZ8CraOOT4w8JqXFmtspxDfYiin7OWxN6TNTid+O\nyM/dwDyszKYBoOZ2YX8E7gRaAv+s7heCWgKKrgsuAIZik8baAD9EVwzEq2XU/cso3TiS2LyDffay\nsc/i5dhnU2qmAdA4cr8hdnWfPpeJWQBE1pblWmC+w1jidhlWYz2AjWBLov7tLqwBtwG4OPmhpbyn\nscvt1mEfimprg1IhTWD0ThvsSqq1wIcon/F6FtgOfIt9b16HXVH1Mil6GaiIiIiIiIiIiIiIiIiI\niIiIiIiIiIhInPYn8LuTsdVsweasiIhICtkXsNcR8V1Ql4IQcSUDGIctS/A+MCTyeC1gIrAem2W5\nGBgY+bcC4Dxs/fWjsA1OnklaxCIikpDiv9wHYl/wGUBzYAu2wNYg7EsfbCmNL4ABkePoTYx0BiAp\nQ2cAIqV1AmZgS+n+E1gO/BRbunhm5Dm7sC99kZSmAUCktCIq36VKu1dJWtEAIFLaSmyp51rYRiUX\nAG8Bb2DloQysBJRbye8fIvn7bIjUiD6oIqZ496R5wM+wpbOLgJFYKWgO0A34O7b87nvAlxW8ziSs\nefwuMMzfkEVEJFkaRn4eh+0J0NxhLCIJ0xmASOwWYZts1MX2WK52yz0RERERERERERERERERERER\nERERERERj/0/W9v3d979EGYAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x = np.arange(-10, 10, 0.1)\n",
"y = [good_xentropy(i, 1) for i in x]\n",
"plt.plot(x, y)\n",
"plt.grid(); plt.xlabel(\"logit\"); plt.ylabel(\"Cross-Entropy\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. Logistic Regression penalizes you linearly when you are on the wrong side of the hyperplane.\n",
"2. Logistic Regression doesn't penalizes you when you are on the right side of the hyper-plane but far away. (Not sensitive to outliers)\n",
"3. This is why we should use logistic regression instead of linear regression for classification (comes at the cost of not having a closed form solution)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Support Vector Machines (SVM):\n",
"\n",
"- The logistic regression cost function is very similar to the cost function of SVM. SVM only considers the points that are close to the hyperplane (support vectors) and ignores the rest of the points:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
""
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEPCAYAAABLIROyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF6hJREFUeJzt3XmUXHWVwPFv2EQj2qKShIGhEQRXbPHgAYGZZmcYNLgl\nMA7S6nA8AoKDAwRU0HGJ4HEmLMYzCLGjCAyLwSU6CmIhREcWCQEUBYdmwBjAACocHYL0/PGrSiqV\nTnctv1e/t3w/5/TpfpXq7ss9Rf363d+774IkSZIkSZIkSZIkSZIkSZK0nkXAw8CdTY99DvgFcAfw\ndeCFCeKSJGVsX+D1rL8AHARsUv/6s/UPSVICm0z9lK7dCDze8ti1wLP1r38KbJfh75ckTSLLBWAq\n7wW+k/D3S1KlpVoAPgI8DVya6PdLUuVtluB3jgCHAQds7Anbbrvt+MqVK/sWkCSVxK+Bndt9cr/P\nAA4FTgFmA3/e2JNWrlzJ+Pi4H5E+zjrrrOQxlOXDXJrPPH8AO3XyhpzlAnAZ8GNgV+BBQs3/fOD5\nhM3g24GFGf5+1Y2NjaUOoTTMZVzmM60sS0BHTfDYogx/nySpAymvAlKfjIyMpA6hNMxlXOYzrWmp\nA9iI8Xo9S5LUpmnTpkEH7+ueAVRArVZLHUJpmMu4zGdaLgCSVFGWgCSpJCwBSZLa4gJQAdZZ4zGX\ncZnPtFwAJKmi3AOQpJJwD0CS1BYXgAqwzhqPuYzLfKblAiBJFeUegCSVhHsAkqS25HYB8Dbh8Vhn\njcdcxmU+08rtAnDoofC736WOQpLKK7d7AKedNk6tBj/4AUyfnjocScq/TvcAcrsAPPvsOCMj4Szg\nmmtg881ThyRJ+VaaTeBp0+Cii2B8HN7//vBZ3bHOGo+5jMt8ppXbBQDCX/1XXgl33QUf/WjqaCSp\nXHJbAmruA3j0Udh7bzjxRDjhhIRRSVKOdVoC2iy7UOJ56Uvhe9+DffaBGTPgne9MHZEkFV+uS0DN\ndtwRli6F448Hy4adsc4aj7mMy3ymVZgFAGBoCC6/HObMgRUrUkcjScVWiD2AVldcASefDDfdBIOD\n/QtKkvKslHsArebMgVWrQrfwTTfBS16SOiJJKp5ClYCanXgiHHEEHH44PPVU6mjyzTprPOYyLvOZ\nVmEXAID582HXXcMZwZo1qaORpGLJcg9gEfD3wCPAa+uPbQ38J7ADMAbMAZ6Y4HvbngewZg3Mng0z\nZ8LFF4cOYkmqojzdCuLLwKEtj80DrgV2AX5QP+6J3cKS1J0sF4AbgcdbHnsLsLj+9WLgiBi/aPr0\n0CNw5ZVwwQUxfmK5WGeNx1zGZT7T6vdVQDOAh+tfP1w/jsJuYUnqTNYV80HgW6zbA3gceFHTvz9G\n2Bdo1fVM4OXL4eCDQ6/A8HBXP0KSCinvfQAPAzOBVcAswgbxhEZGRhisd3kNDAwwNDTEcP0dvXHa\nONHx0BDMm1fjiCPgRz8aZrfdJn++xx577HFRj2u1GqOjowBr3y870e8zgHOA1cDZhA3gASbeCO76\nDKDBbuF1arXa2hePemMu4zKfceXpDOAy4G+BlwAPAmcCnwWuAN7HustAM2G3sCRNLq9Xzfd8BtAw\nb164e6izhSWVXWlmAsdaAMbHcbawpErIUyNYLjhb2GutYzKXcZnPtEq/AIDdwpI0kdKXgJo5W1hS\nmeXpKqDcsVtYktapRAmoWRVnC1tnjcdcxmU+06rcAgDOFpYkqNgeQCu7hSWViXsAHbBbWFKVVbIE\n1KwKs4Wts8ZjLuMyn2lVfgEAZwtLqqZK7wE0c7awpKLzVhBdsltYUtW4ADQp62xh66zxmMu4zGda\nlb4KaCJ2C0uqirxWuvu+B9DK2cKSisY9gEjsFpZUdi4Ak9h//7AXcNhhMDaWOpruWWeNx1zGZT7T\ncg9gCnYLSyor9wDa5GxhSXnnTOCMOFtYUt65CZyRIs8Wts4aj7mMy3ym5QLQAbuFJZWJJaAuOFtY\nUh45D6APmruFZ86Ed7wjdUSS1DlLQF1qzBY+7rj8zxa2zhqPuYzLfKblAtADu4UlFZl7ABE4W1hS\nHrgHkIDdwpKKKFUJ6HTgbuBO4FLgOYniiCbPs4Wts8ZjLuMyn2mlWAAGgWOB3YHXApsCRyaII7rG\nbOG5c50tLCn/UuwBbA38BNgT+COwBDgXuK7pOYXaA2jmbGFJqRThVhCPAZ8H/hdYCTzB+m/+hWa3\nsKSiSLEJvBPwIUIp6PfAlcC7gK81P2lkZITB+iU1AwMDDA0NMVwfzdWoG+b1+JZbapxxBpx66jCz\nZsFrXpM2ngULFhQqf3k+bq5Z5yGeoh+bz97zNzo6CrD2/bITKQoUc4GDgH+qHx9NKAcd3/ScwpaA\nmt1/f+gWPvfctN3CtVpt7YtHvTGXcZnPuIpwO+jXEf7a3wP4MzAK3Ax8oek5pVgAwNnCkvqnCHsA\ndwBfAW4FGv2zFyaIoy/sFpaUV+0sAJtm8HvPAV5NuAz0GKDUF02mni3cXGdVb8xlXOYzrXY2ge8F\nrga+DPw823DKy25hSXnTTq3oBYRGrRHC2cAi4DLgD9mFVZ49gFbOFpaUlaw3gYcJG7gvIly++Ung\nvg5/RjtKuwA0ZguvXg1LljhbWFI8WWwCbwbMBq4BFhCauF4GfAv4TuchVltjtvCzz/ZvtrB11njM\nZVzmM612FoBfERaAc4Ah4N+AVcBVwPeyC6287BaWlAftnCpsRbhnTz+VtgTUzNnCkmLKogS0DaHc\n8zvgUeAbhBKQetSYLTx/Plx1VepoJFVNOwvApcAVwCxgW8Lm72VZBlUl/ZgtbJ01HnMZl/lMq50F\n4LnAVwnNWmuAS4AtswyqauwWlpRCO7Wiswm3bG781T+XcBnoOfXjxzKIqxJ7AK2cLSypF1n0AYwB\nG3s3Hieb/YBKLgAA550HCxfaLSypc1lsAg8CO27kw83gyLKYLWydNR5zGZf5TKudBWAL4CTC/YCu\nAj4I2L+aIWcLS+qHdk4VLiZ0Ay+uP/9o4BnWDXTJQmVLQA3OFpbUqSz2AFYAu7XxWEyVXwAglID2\n2w8OOgg+/enU0UjKuyz2AJ4Bdm463qn+mDI2fXroEbjyyjBPoFvWWeMxl3GZz7TamQdwCnA9cH/9\neBB4T1YBaX2NbuF99gnloJSzhSWVy1SnCpsSNoAXArvWH/slYZZvliwBtXC2sKSpZLEHcAthgHs/\nuQBM4Prr4cgj4brrYLcsd2AkFVIWewA3ARcA+wK7A2+of1afdTtb2DprPOYyLvOZVjt7AK8ndPz+\na8vj+8UPR1NxtrCkWNo5VXgZ8D9tPBaTJaApOFtYUqss9gB+xoYln9sIpaCsuABMwdnCklrF3AN4\nJfB2YAB4W/3rtwEjeDvo5DqZLWydNR5zGZf5TGuyBWAX4M3AC+ufD69/3h04NvvQNBVnC0vqRTun\nCm8Cfpx1IC0sAXXA2cKSoPMSUDtXAd0HfITQAdx4/jjw3g5jU0bsFpbUjXb6AL4BvAC4Flja9KEc\nmWy2sHXWeMxlXOYzrXbOAJ4LnBb59w4AFwGvZt3ZxH9H/h2V0zxb2G5hSVNpp1b0KeAnxP2rfzFw\nA7CIsAhNB37f9O/uAfSgMVt42TLYYYfU0Ujqlyz6AJ4Engc8DTTmU40TykLdeCFwO5OPk3QB6JGz\nhaXqyeJeQM+vP29LYKv6R7dv/hBmCT8KfJnQZPYlwgKjiJpnC3/3u7XU4ZSGNeu4zGdak+0B/CNw\nSf3rvYFlTf92AuEGcd3+zt3rP+MWYAEwDziz+UkjIyMMDg4CMDAwwNDQEMP1+yA3XjQeT348f/4w\nIyPw4Q8vZ/PN4cAD8xWfxx573NtxrVZjdHQUYO37ZScmO1W4nXAjuNavJzruxEzCnsKO9eN9CAvA\n4U3PsQQUibOFperIogQU2yrgQUKnMcCBwN0J4qiE5m7hj30sdTSS8iTFAgDwQeBrwB2E4fKfSRRH\nJdxyS42lS8PVQb3MFpY169jMZ1qT7QG8Ariz/vVOTV83jntxB/2fMlZpdgtLajVZrWhwiu8dixfG\nBtwDyIizhaXyyqIPoNnhwLc7/J5uuABkyNnCUjllvQn8yQ6frxxorbM2zxZ+4IE0MRWVNeu4zGda\n7dwLSCXUmC18yCF2C0tV1WkJ6I3AzVkE0sISUJ84W1gqjyxKQHNYd+uHQ4AlbDgjWAU1fz7suivM\nnRuaxiRVRzsLwMeAPxA6dg8ALga+mGVQimuyOmsns4VlzTo285lWOwvAX+qfDyfcuO3bwBaZRaS+\ns1tYqqZ2akVLgd8ABxHu//Nn4KfA6zKMyz2ABJwtLBVbFn0A04FDgRXAvcAs4LXA97uIr10uAInc\nf3/oFj73XLuFpaLJYhN4JuEs4F5gP8KmcD+uBFIkndRZJ5stLGvWsZnPtNpZAL4OPAPsDPwHsB1w\naZZBKa3m2cIrVqSORlJW2jlVaNz7/1TgT8D59DYPoB2WgHLA2cJSsXRaAmqnE/hp4B+AdwNvrj+2\neceRqXDsFpbKrZ0S0HuBvYBPA/cThrl/NcugFFcvddbm2cJPPRUvpqKyZh2X+UyrnQXgbuBfgLuA\n1xCmeZ2dZVDKF7uFpXJqp1Y0DCwGGveN/GvgGOCGjGIC9wByx9nCUv5l0QfwM+Ao4Jf1412Ay8n2\nfkAuADn01FOw335hoMynPpU6GkmtsugD2Ix1b/4Av8LbSBdKrDrr9OlUfrawNeu4zGda7byR3wZc\nBFxCWFneBdyaZVDKL2cLS+XRzqnCc4ATgL3rxzcCC4H/yyooLAHlnrOFpfyJvQewGeHqn1f0EFM3\nXAAKwNnCUr7E3gN4hlD/tw+0wLKqs1ZxtrA167jMZ1rt7AFsTegFuBlotAKNA2/JKigVh93CUnFN\ndqrwcmAGsGnL8/YFfkvYGM6KJaCCcbawlF7MPYClwOmEOQDNdiPcFuLNG3xHPC4ABTM+DiMjsHo1\nLFkSpoxJ6q+YewAz2PDNn/pjO3YWllLqR521KrOFrVnHZT7TmmwBGJjk37aMHYiKz9nCUrFMdqpw\nOXA9cGHL48cCBwJze/zdmxIayh5iw3KSJaACc7awlEbMeQAfApYQOn9vqz/2BkJj2Fu7jK/ZScDP\nga0i/CzliN3CUjFMVgJaBbwJ+AQwRpgF8AlgT8JVQL3YDjiMcCWR95XMWIo6a1lnC1uzjst8pjVV\nH8A4oQx0feTf++/AKcALIv9c5UjzbGG7haX8SXFXz8OBRwhzhYc39qSRkREGBwcBGBgYYGhoiOH6\nTWcafzV43N5x47EUv3///eEDH6hxwAFw663D7LBD+nz0cjw8PJyreIp+bD57O67VaoyOjgKsfb/s\nRIryy2eAowm3mdiScBZwNWHmcIObwCVz3nmwcKHdwlKWspgHENsZwPaEXoIjCeWld0/6HepJ4y+G\nlMoyWzgPuSwT85lWigWglX/qV4SzhaV8yesVOJaASsrZwlJ2ilACUoXZLSzlhwtABeStzlrk2cJ5\ny2XRmc+0HO6uJOwWltLLawXWPYCKcLawFI97ACqURrfw3LmwYqKbj0vKjAtABeS9zrr//nD++cWY\nLZz3XBaN+UzLPQDlgrOFpf5zD0C54mxhqXsxZwKn5AJQUc2zha+5BjbzHFVqm5vA2kCR6qx5ny1c\npFwWgflMywVAudPoFr7zTruFpSxZAlJuOVtY6kzMmcBSUnYLS9myBFQBRa6z5m22cJFzmUfmMy0X\nAOWe3cJSNtwDUGFccQWcfDIsWwY77JA6Gil/3ANQadktLMVlCagCylRnTT1buEy5zAPzmZYLgAqn\nebbwM8+kjkYqLvcAVEiN2cKzZoXOYWcLS94KQhVht7DUOxeACihrnTXFbOGy5jIV85mWVwGp0OwW\nlrqX18qpewDqiLOFJfcAVFF2C0udcwGogKrUWfsxW7gquewX85mWewAqFbuFpfa5B6BScrawqqgI\nM4G3B74CbAOMAxcC57U8xwVAPXG2sKqoCJvAa4B/Bl4N7AkcD7wyQRyVUcU6a1azhauYyyyZz7RS\nLACrgOX1r58EfgFsmyAOlZzdwtLkUu8BDAI3EM4Gnmx63BKQonG2sKqiSPMAng9cBZzE+m/+UlR2\nC0sTS7UAbA5cDVwCXDPRE0ZGRhgcHARgYGCAoaEhhustno26ocftHS9YsMD8AUuXDnPwwfDQQzWG\nhrr7ec0169T/PWU4Np+95290dBRg7ftlJ1KUgKYBi4HVhM3giVgCiqhWq6198VTd9dfDUUfBtdfC\nbrt1/v3mMi7zGVcRLgPdB/gRsIJwGSjA6cB/NT3HBUCZcbawyqoIewA34S0olJDdwlLgG3EFNNdZ\nFXQ7W9hcxmU+03IBUGU5W1hVl7oPYGPcA1BfOFtYZVKEW0FIuWG3sKrMBaACrLNOrpPZwuYyLvOZ\nlvdIlLBbWNWU14qnewBKwtnCKjL3AKQeOFtYVeICUAHWWTsz2WxhcxmX+UzLPQBpAs3dwsuWwYtf\nnDoiKT73AKRJzJsHN9wA113nbGHlXxFuBtcOFwDlgrOFVSRuAmsD1lm71zpb+Ic/rKUOqVR8babl\nAiBNoblbeNGi1NFI8VgCktrUmC180klw/PGpo5E2VIR5AFIhNXcLz5hht7CKzxJQBVhnjeeBB2os\nXQrHHQemtXe+NtNyAZA6ZLewysI9AKlLzhZW3rgHIPWJ3cIqOktAFWCdNZ7WXHY7W1iBr820XACk\nHs2fD7vs4mxhFY97AFIEzhZWHngrCCkBZwuriFwAKsA6azyT5bJ5tvAXvtC/mIrM12ZaXgUkRWS3\nsIokr5VK9wBUaM4WVgruAUg5YLewiiDVAnAocA9wL3BaohgqwzprPJ3kcrLZwgp8baaVYgHYFLiA\nsAi8CjgKeGWCOCpj+fLlqUMojU5zOWcOnHpq6BZevTqjoArM12ZaKRaANwL3AWPAGuByYHaCOCrj\niSeeSB1CaXSTS7uFN87XZlopFoC/Ah5sOn6o/phUWnYLK49SXAbq5T19NjY2ljqE0ug2l43ZwrNn\nw157wcyZceMqqttvH+O221JHUXx77AFnntn596W4DHRP4OOEPQCA04FngbObnnMfsFN/w5Kkwvs1\nsHPqICazGSHIQWALYDluAktSZfwd8EvCX/qnJ45FkiRJUgrvBO4G/gLs3vJvpxOaxu4BDu5zXGXw\nccLVVrfXPw6d9NnaGBsY4xoDVhBekzenDaVwFgEPA3c2PbY1cC3wK+D7wECCuLr2CmAX4IesvwC8\nirBPsDlh3+A+vIVFp84CTk4dRMFtSnjtDRJei+5d9e5+wpuWOrcv8HrWXwDOAU6tf30a8Nmpfkie\n3kjvIaxcrWYDlxGaxsYI/xO+sX9hlUZeb/xXFDYwZsPXZXduBB5veewtwOL614uBI6b6IXlaADZm\nW0L5osHGse58ELgDuJiCnRrmhA2M8Y0D1wG3AscmjqUMZhDKQtQ/z5jqG/rdCHYtMFELzBnAtzr4\nOTaTbWhjuf0I8EXgX+vHnwQ+D7yvT3GVha+5+PYGfgu8lPD6vYfwl616N04br9l+LwAHdfE9vwG2\nbzrerv6Y1tdubi+is8VWQevrcHvWPzNV535b//wosIRQZnMB6N7DhD8CVwGzgEem+oa8loCa64Lf\nBI4kNI3tCLwcrxjo1Kymr9/K+htHas+thNfeIOG1OJfw2lR3ngdsVf96OuHqPl+XvfkmcEz962OA\naxLG0rG3EmqsfyKsYN9t+rczCBtw9wCH9D+0wvsK4XK7Owgviilrg5qQDYzx7Ei4kmo5cBfms1OX\nASuBpwnvm+8hXFF1HQW9DFSSJEmSJEmSJEmSJEmSJEmSOvRkD9/7JcLdbCH0rEiSCuSPOfs5Uuby\neisIKZVpwOcItyVYAcypP74JsBD4BaHLcinw9vq/1YA3EO6//lzCgJOv9i1iSVJPGn+5v53wBj8N\n2AZ4gHCDrXcQ3vQh3ErjMeBt9ePmIUaeAagwPAOQ1rcPcCnhVrqPADcAexBuXXxF/TkPE970pUJz\nAZDWN87Gp1Q5vUql4gIgre9Gwq2eNyEMKvkb4KfAMkJ5aBqhBDS8ke9fQ//nbEhd8YUqBY3pSUuA\nvQi3zh4HTiGUgq4GDgB+Trj97s+A30/wcy4kbB7fBhydbciSpH6ZXv/8YsJMgG0SxiL1zDMAqX3f\nJgzZ2IIwY3nKkXuSJEmSJEmSJEmSJEmSJEmSFNn/A1k3rKuen9PiAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def svm_cost(x):\n",
" return - x + 1 if x < 1 else 0\n",
"x = np.arange(-10, 10, 0.1)\n",
"y = [svm_cost(i) for i in x]\n",
"plt.plot(x, y)\n",
"plt.grid(); plt.xlabel(\"logit\"); plt.ylabel(\"Cross-Entropy\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cross Entropy in TensorFlow"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cost_batch = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, targets=Y)\n",
"cost = tf.reduce_mean(cost_batch)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TensorShape([Dimension(None), Dimension(1)])\n",
"TensorShape([])\n"
]
}
],
"source": [
"print logits.get_shape()\n",
"print cost.get_shape()"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"norm_w = tf.nn.l2_loss(W)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Momentum Optimizer\n",
"\"This is logistic regression on noisy moons dataset from sklearn which shows the smoothing effects of momentum based techniques (which also results in over shooting and correction). The error surface is visualized as an average over the whole dataset empirically, but the trajectories show the dynamics of minibatches on noisy data. The bottom chart is an accuracy plot.\" (Image by Alec Radford)\n",
"\n",
"![Momentum](http://2.bp.blogspot.com/-q6l20Vs4P_w/VPmIC7sEhnI/AAAAAAAACC4/g3UOUX2r_yA/s1600/s25RsOr%2B-%2BImgur.gif =100x20)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"optimizer = tf.train.MomentumOptimizer(learning_rate=1.0, momentum=0.99)\n",
"# optimizer = tf.train.GradientDescentOptimizer(learning_rate=1.0)\n",
"train_op = optimizer.minimize(cost)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Compute Accuracy"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#a hack for binary thresholding\n",
"pred = tf.greater(output, 0.5)\n",
"pred_float = tf.cast(pred, \"float\")\n",
"\n",
"#accuracy\n",
"correct_prediction = tf.equal(pred_float, Y)\n",
"accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating a session"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"sess = tf.InteractiveSession()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Intitializing Variables"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"init = tf.initialize_all_variables()\n",
"sess.run(init)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Training"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Epoch:0001, cost=1.221206784, Train Accuracy=0.6900, Eval Accuracy=0.6650, Norm of Weights=78.0369\n",
"Epoch:0051, cost=0.108576626, Train Accuracy=0.9925, Eval Accuracy=0.9525, Norm of Weights=21902.0879\n",
"Epoch:0101, cost=0.039827708, Train Accuracy=0.9962, Eval Accuracy=0.9400, Norm of Weights=21526.6953\n",
"Epoch:0151, cost=0.016990982, Train Accuracy=0.9975, Eval Accuracy=0.9400, Norm of Weights=21287.8965\n",
"Epoch:0201, cost=0.008619667, Train Accuracy=0.9987, Eval Accuracy=0.9375, Norm of Weights=21630.7266\n",
"Epoch:0251, cost=0.006221656, Train Accuracy=1.0000, Eval Accuracy=0.9375, Norm of Weights=22097.6465\n",
"Epoch:0301, cost=0.005005265, Train Accuracy=1.0000, Eval Accuracy=0.9375, Norm of Weights=22532.3867\n",
"Epoch:0351, cost=0.004219066, Train Accuracy=1.0000, Eval Accuracy=0.9375, Norm of Weights=22931.1543\n",
"Epoch:0401, cost=0.003665265, Train Accuracy=1.0000, Eval Accuracy=0.9375, Norm of Weights=23299.3828\n",
"Epoch:0451, cost=0.003253295, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=23642.1973\n",
"Epoch:0501, cost=0.002933940, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=23963.8164\n",
"Epoch:0551, cost=0.002679895, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=24267.5586\n",
"Epoch:0601, cost=0.002475345, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=24556.2324\n",
"Epoch:0651, cost=0.002309947, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=24832.2344\n",
"Epoch:0701, cost=0.002175402, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=25097.5879\n",
"Epoch:0751, cost=0.002064377, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=25353.9160\n",
"Epoch:0801, cost=0.001970768, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=25602.5332\n",
"Epoch:0851, cost=0.001890046, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=25844.4199\n",
"Epoch:0901, cost=0.001819121, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=26080.3594\n",
"Epoch:0951, cost=0.001755936, Train Accuracy=1.0000, Eval Accuracy=0.9350, Norm of Weights=26310.9219\n",
"Epoch:1001, cost=0.001699073, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=26536.5977\n",
"Epoch:1051, cost=0.001647520, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=26757.8105\n",
"Epoch:1101, cost=0.001600507, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=26974.9258\n",
"Epoch:1151, cost=0.001557441, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=27188.2422\n",
"Epoch:1201, cost=0.001517841, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=27398.0293\n",
"Epoch:1251, cost=0.001481280, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=27604.5820\n",
"Epoch:1301, cost=0.001447390, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=27808.0020\n",
"Epoch:1351, cost=0.001415835, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=28008.5566\n",
"Epoch:1401, cost=0.001386318, Train Accuracy=1.0000, Eval Accuracy=0.9325, Norm of Weights=28206.3594\n",
"Epoch:1451, cost=0.001358589, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=28401.5312\n",
"Epoch:1501, cost=0.001332433, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=28594.2266\n",
"Epoch:1551, cost=0.001307678, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=28784.4531\n",
"Epoch:1601, cost=0.001284176, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=28972.3613\n",
"Epoch:1651, cost=0.001261803, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=29158.0625\n",
"Epoch:1701, cost=0.001240452, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=29341.5488\n",
"Epoch:1751, cost=0.001220033, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=29522.9199\n",
"Epoch:1801, cost=0.001200472, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=29702.2598\n",
"Epoch:1851, cost=0.001181695, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=29879.6602\n",
"Epoch:1901, cost=0.001163654, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=30055.0801\n",
"Epoch:1951, cost=0.001146289, Train Accuracy=1.0000, Eval Accuracy=0.9300, Norm of Weights=30228.6738\n"
]
}
],
"source": [
"for epoch in range(2000):\n",
" for i in xrange(8):\n",
" x_batch = x_train[i * 100: (i + 1) * 100]\n",
" y_batch = t_train[i * 100: (i + 1) * 100]\n",
" cost_np, _ = sess.run([cost, train_op],\n",
" feed_dict={X: x_batch, Y: y_batch})\n",
" #Display logs per epoch step\n",
" if epoch % 50 == 0:\n",
" cost_train, accuracy_train = sess.run([cost, accuracy],\n",
" feed_dict={X: x_train, Y: t_train})\n",
" cost_eval, accuracy_eval, norm_w_np = sess.run([cost, accuracy, norm_w],\n",
" feed_dict={X: x_eval, Y: t_eval}) \n",
" print (\"Epoch:%04d, cost=%0.9f, Train Accuracy=%0.4f, Eval Accuracy=%0.4f, Norm of Weights=%0.4f\" %\n",
" (epoch+1, cost_train, accuracy_train, accuracy_eval, norm_w_np))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## $L_2$ Regularization\n",
"\n",
"As you can see when the data is linearly separable, the norm of W goes to infinity! (Can you explain why?)\n",
"\n",
"Add L2 regularization to the above code so as to prevent this from happening (only one line of code! Thanks to awesome TensorFlow!)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# Non-Linear Feature Space\n",
"\n",
"\n",
"\n",
""
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python [default]",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 0
}